How do I deploy a rosdep mirror?

asked 2019-11-20 12:09:16 -0500

rubicks
193 ●19 ●30 ●30

updated 2019-11-20 16:13:44 -0500

Github is starting to throttle my requests to https://raw.githubusercontent.com/ros... . How do I deploy a mirror of such a package index and tell rosdep to consume it?

Edit: responding the obvious question; it's probably in the neighborhood of 5000 invocations daily from my employer's office. I've already taken steps eliminate the ones I make during sundry container initialization, but there are some that I can't eliminate --- notably the one issued by bloom-release when I bloom my projects.

edit retag flag offensive close merge delete

Comments

Just to get some sense of scale here: how many package (re)builds are you running? Or how many packages are you building per time-unit?

I'm not aware of build.ros.org using any mirror or cache for ros/rosdistro specifically, so I'm wondering what it would take for Github to start throttling you.

gvdhoorn ( 2019-11-20 13:58:31 -0500 )edit

Wow, that's a lot of calls. Can you confirm which operations are problematic, is it the rosdep updates that are failing? Those should not be necessary to call that often as the rosdep database doesn't change that often. Most rosdistro operations are designed to use the cache.

It would be possible to add an enhancement to rosdep to use a github token to do authenticated fetching with a much higher limit. And bloom could also be enhanced to do the same as well as pass the credentials through.

For reference this implies that this is governed by the unauthenticated API rate limit of 60/hour, Authenticating will get you up to 5000/hour. GitHub API rate limiting

tfoote ( 2019-11-20 17:24:26 -0500 )edit

it's probably in the neighborhood of 5000 invocations daily from my employer's office

can you say anything about what you're actually doing that requires that many rosdep updates?

there are some that I can't eliminate --- notably the one issued by bloom-release when I bloom my projects.

5000 bloom-releases per day (?).

gvdhoorn ( 2019-11-21 09:37:26 -0500 )edit

There's about 50 invocations to bloom-release daily. I've been able to eliminate all other invocations to rosdep update. Now, I rosdep update as root during the construction of a build environment and then copy ~root/.ros to ~luser when I wish to build catkin projects.

rubicks ( 2019-11-21 11:11:45 -0500 )edit

add a comment

2 Answers

Sort by » oldest newest most voted

answered 2019-11-20 14:11:13 -0500

tfoote

58457 ●128 ●550 ●526 http://www.ros.org/

updated 2019-11-22 16:52:45 -0500

As a direct answer to your question you can setup your own rosdep sources. See the documentation here: https://docs.ros.org/independent/api/...

But I have to agree with @gvdhoorn that we have used this a lot and have not experienced throttling from github on our CI setups. How often are you calling rosdep update?

I put together a quick Dockerfile to demo how to use a local copy:

FROM ros
# pull a copy of the rosdistro
RUN mkdir -p /cache && cd /cache && git clone https://github.com/ros/rosdistro.git -b master --depth 1
# Update it to self reference
RUN sed -i s#https://raw.githubusercontent.com/ros/rosdistro/master/#file:///cache/rosdistro/# /cache/rosdistro/rosdep/sources.list.d//20-default.list
# Tell the local environment to use the local instance of rosdistro
ENV ROSDISTRO_INDEX_URL="file:///cache/rosdistro/index-v4.yaml"
# Tell rosdep where to look for sources
ENV ROSDEP_SOURCE_PATH=/cache/rosdistro/rosdep/sources.list.d
# Clear system sources for clarity. They will be overridden anyway, but to show for this demo they're not being used.
RUN rm -rf /etc/ros/rosdep

# Show rosdep working using local resources
CMD rosdep update

You can host the clone of the rosdistro anywhere you want. it's convenient for this demo to be on the filesystem.

edit flag offensive delete link

Comments

I can certainly replace the default list...

https://github.com/ros/rosdistro/blob/master/rosdep/sources.list.d/20-default.list

...with something that references the same gbpdistro and yaml content under file:/// URIs instead of https:// URIs. But when I try that trick with the DEFAULT_INDEX_URL...

https://github.com/ros-infrastructure...

...it errors out looking for non-existent files on my host. Any ideas what's happening there?

rubicks ( 2019-11-20 17:26:34 -0500 )edit

What are you doing to change the DEFAULT_INDEX_URL? What are you running? And what are the errors?

tfoote ( 2019-11-20 17:35:05 -0500 )edit

Here's a Dockerfiledemonstrating what I mean: https://hastebin.com/raw/ocitiqujeg

The error produced is as follows:

    ERROR: error loading sources list:
    <urlopen error <urlopen error [Errno 2] No such file or directory: '/var/lib/rosdep/crystal/distribution.yaml'>(file:///var/lib/rosdep/crystal/distribution.yaml)>

rubicks ( 2019-11-21 11:07:26 -0500 )edit

I saw your hastebin posting this morning, but now it' won't load on a heroku error

Application error An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command heroku logs --tail

tfoote ( 2019-11-21 18:27:45 -0500 )edit

Shucks. Try this one: https://gist.github.com/rubicks/0a9e2...

rubicks ( 2019-11-22 09:00:27 -0500 )edit

I updated my answer with an example of how to use rosdep using a local copy of the rosdistro in a Dockerfile

tfoote ( 2019-11-22 16:53:34 -0500 )edit

That will certainly work for my purposes --- thanks very much.

rubicks ( 2019-11-25 08:42:11 -0500 )edit

add a comment

answered 2019-12-13 11:27:07 -0500

rubicks
193 ●19 ●30 ●30

As a counterpart, I present my maximally sneaky and invasive implementation:

# frozen-rosdep.dockerfile
FROM ros
ENV ROSDISTRO_INDEX_URL="file:///etc/ros/index-v4.yaml"
RUN set -eu \
  && apt -y update \
  && apt -y install curl \
  && curl -fsSL https://github.com/ros/rosdistro/archive/master.tar.gz \
       | tar -C /etc/ros --strip-components 1 -xzf- \
  && sed -i s,https://raw.githubusercontent.com/ros/rosdistro/master,file:///etc/ros,g \
       /etc/ros/rosdep/sources.list.d/20-default.list \
  && rosdep update --verbose \
  && true

edit flag offensive delete link

add a comment

How do I deploy a rosdep mirror?

Comments

2 Answers

Comments

Question Tools

Stats

Related questions

How do I deploy a rosdep mirror? edit

Comments

2 Answers

Comments

Question Tools

Stats

Related questions

How do I deploy a rosdep mirror?