Ask Your Question
1

How do I deploy a rosdep mirror?

asked 2019-11-20 12:09:16 -0600

rubicks gravatar image

updated 2019-11-20 16:13:44 -0600

Github is starting to throttle my requests to https://raw.githubusercontent.com/ros... . How do I deploy a mirror of such a package index and tell rosdep to consume it?

Edit: responding the obvious question; it's probably in the neighborhood of 5000 invocations daily from my employer's office. I've already taken steps eliminate the ones I make during sundry container initialization, but there are some that I can't eliminate --- notably the one issued by bloom-release when I bloom my projects.

edit retag flag offensive close merge delete

Comments

Just to get some sense of scale here: how many package (re)builds are you running? Or how many packages are you building per time-unit?

I'm not aware of build.ros.org using any mirror or cache for ros/rosdistro specifically, so I'm wondering what it would take for Github to start throttling you.

gvdhoorn gravatar imagegvdhoorn ( 2019-11-20 13:58:31 -0600 )edit

Wow, that's a lot of calls. Can you confirm which operations are problematic, is it the rosdep updates that are failing? Those should not be necessary to call that often as the rosdep database doesn't change that often. Most rosdistro operations are designed to use the cache.

It would be possible to add an enhancement to rosdep to use a github token to do authenticated fetching with a much higher limit. And bloom could also be enhanced to do the same as well as pass the credentials through.

For reference this implies that this is governed by the unauthenticated API rate limit of 60/hour, Authenticating will get you up to 5000/hour. GitHub API rate limiting

tfoote gravatar imagetfoote ( 2019-11-20 17:24:26 -0600 )edit

it's probably in the neighborhood of 5000 invocations daily from my employer's office

can you say anything about what you're actually doing that requires that many rosdep updates?

there are some that I can't eliminate --- notably the one issued by bloom-release when I bloom my projects.

5000 bloom-releases per day (?).

gvdhoorn gravatar imagegvdhoorn ( 2019-11-21 09:37:26 -0600 )edit

There's about 50 invocations to bloom-release daily. I've been able to eliminate all other invocations to rosdep update. Now, I rosdep update as root during the construction of a build environment and then copy ~root/.ros to ~luser when I wish to build catkin projects.

rubicks gravatar imagerubicks ( 2019-11-21 11:11:45 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
2

answered 2019-11-20 14:11:13 -0600

tfoote gravatar image

updated 2019-11-22 16:52:45 -0600

As a direct answer to your question you can setup your own rosdep sources. See the documentation here: https://docs.ros.org/independent/api/...

But I have to agree with @gvdhoorn that we have used this a lot and have not experienced throttling from github on our CI setups. How often are you calling rosdep update?

I put together a quick Dockerfile to demo how to use a local copy:

FROM ros
# pull a copy of the rosdistro
RUN mkdir -p /cache && cd /cache && git clone https://github.com/ros/rosdistro.git -b master --depth 1
# Update it to self reference
RUN sed -i s#https://raw.githubusercontent.com/ros/rosdistro/master/#file:///cache/rosdistro/# /cache/rosdistro/rosdep/sources.list.d//20-default.list
# Tell the local environment to use the local instance of rosdistro
ENV ROSDISTRO_INDEX_URL="file:///cache/rosdistro/index-v4.yaml"
# Tell rosdep where to look for sources
ENV ROSDEP_SOURCE_PATH=/cache/rosdistro/rosdep/sources.list.d
# Clear system sources for clarity. They will be overridden anyway, but to show for this demo they're not being used.
RUN rm -rf /etc/ros/rosdep

# Show rosdep working using local resources
CMD rosdep update

You can host the clone of the rosdistro anywhere you want. it's convenient for this demo to be on the filesystem.

edit flag offensive delete link more

Comments

I can certainly replace the default list...

https://github.com/ros/rosdistro/blob/master/rosdep/sources.list.d/20-default.list

...with something that references the same gbpdistro and yaml content under file:/// URIs instead of https:// URIs. But when I try that trick with the DEFAULT_INDEX_URL...

https://github.com/ros-infrastructure...

...it errors out looking for non-existent files on my host. Any ideas what's happening there?

rubicks gravatar imagerubicks ( 2019-11-20 17:26:34 -0600 )edit

What are you doing to change the DEFAULT_INDEX_URL? What are you running? And what are the errors?

tfoote gravatar imagetfoote ( 2019-11-20 17:35:05 -0600 )edit

Here's a Dockerfiledemonstrating what I mean: https://hastebin.com/raw/ocitiqujeg

The error produced is as follows:

    ERROR: error loading sources list:
    <urlopen error <urlopen error [Errno 2] No such file or directory: '/var/lib/rosdep/crystal/distribution.yaml'>(file:///var/lib/rosdep/crystal/distribution.yaml)>
rubicks gravatar imagerubicks ( 2019-11-21 11:07:26 -0600 )edit

I saw your hastebin posting this morning, but now it' won't load on a heroku error

Application error An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command heroku logs --tail

tfoote gravatar imagetfoote ( 2019-11-21 18:27:45 -0600 )edit
rubicks gravatar imagerubicks ( 2019-11-22 09:00:27 -0600 )edit

I updated my answer with an example of how to use rosdep using a local copy of the rosdistro in a Dockerfile

tfoote gravatar imagetfoote ( 2019-11-22 16:53:34 -0600 )edit

That will certainly work for my purposes --- thanks very much.

rubicks gravatar imagerubicks ( 2019-11-25 08:42:11 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2019-11-20 12:09:16 -0600

Seen: 57 times

Last updated: Nov 22