As a direct answer to your question you can setup your own rosdep sources. See the documentation here: https://docs.ros.org/independent/api/...
But I have to agree with @gvdhoorn that we have used this a lot and have not experienced throttling from github on our CI setups. How often are you calling rosdep update?
I put together a quick Dockerfile to demo how to use a local copy:
FROM ros
# pull a copy of the rosdistro
RUN mkdir -p /cache && cd /cache && git clone https://github.com/ros/rosdistro.git -b master --depth 1
# Update it to self reference
RUN sed -i s#https://raw.githubusercontent.com/ros/rosdistro/master/#file:///cache/rosdistro/# /cache/rosdistro/rosdep/sources.list.d//20-default.list
# Tell the local environment to use the local instance of rosdistro
ENV ROSDISTRO_INDEX_URL="file:///cache/rosdistro/index-v4.yaml"
# Tell rosdep where to look for sources
ENV ROSDEP_SOURCE_PATH=/cache/rosdistro/rosdep/sources.list.d
# Clear system sources for clarity. They will be overridden anyway, but to show for this demo they're not being used.
RUN rm -rf /etc/ros/rosdep
# Show rosdep working using local resources
CMD rosdep update
You can host the clone of the rosdistro anywhere you want. it's convenient for this demo to be on the filesystem.
Just to get some sense of scale here: how many package (re)builds are you running? Or how many packages are you building per time-unit?
I'm not aware of
build.ros.org
using any mirror or cache forros/rosdistro
specifically, so I'm wondering what it would take for Github to start throttling you.Wow, that's a lot of calls. Can you confirm which operations are problematic, is it the rosdep updates that are failing? Those should not be necessary to call that often as the rosdep database doesn't change that often. Most rosdistro operations are designed to use the cache.
It would be possible to add an enhancement to rosdep to use a github token to do authenticated fetching with a much higher limit. And bloom could also be enhanced to do the same as well as pass the credentials through.
For reference this implies that this is governed by the unauthenticated API rate limit of 60/hour, Authenticating will get you up to 5000/hour. GitHub API rate limiting
can you say anything about what you're actually doing that requires that many
rosdep update
s?5000
bloom-release
s per day (?).There's about 50 invocations to
bloom-release
daily. I've been able to eliminate all other invocations torosdep update
. Now, Irosdep update
asroot
during the construction of a build environment and then copy~root/.ros
to~luser
when I wish to build catkin projects.