ROS service TCP packet structure
Is somewhere in documentation stated the structure of a TCP packet used for persistent service communication? I deduced, that last bytes correspond to service arguments (or response values), however the rest is a undecodable for me.
I want to sniff (tcpdump
) TCP packets sent by ROS service between two nodes. However, I need the detect correct packets from all packets sent by a node, so I think I need to locate bytes, which uniquely identify the service. But where they are and where I can find the signature of the specific service? I tried to locate md5 sum, but it is not there.
Thank you for any hints, I know, I can spend many hours reading the source code to understand in detail, how serialisation works in ROS, but a simple packet structure would be a huge help to me.
Asked by m.bahno on 2021-06-14 08:12:17 UTC
Answers
After some time, I was able (with help of gvdhoorn) to get the structure by myself (I found no clear structure description in documentation). So this is what I found out on the way:
For persistent services, service call TCP packets structure is following (excluding ROS non-related TCP structure):
Request: 4B (payload length) + XB (payload = all service arguments)
Response: 1B (sevice status 'ok' byte - see here) + 4B (payload length) + XB (payload = all service return values)
Unfortunately, there is no service identifier included in those messages, only on initial handshake. However, one option would be to include one more argument (return value), which would be constant identifier. Another option is to find out client communication port and dump only communication on this port. Nevertheless, both approaches have its disadvantages.
Related tips&tricks:
wireshark
is able to recognize ROS TCP packet and read header - usefull for ROS messages or non-persistant services
Asked by m.bahno on 2021-06-16 04:51:05 UTC
Comments
If a posteriori analysis would be good enough for your use-case you could write a Python script with something like caizhengxin/python-libpcap or KimiNewt/pyshark which iterates over all packets and then reconstructs the service exchanges based on the XML-RPC traffic exchanged by the nodes and the master.
That should not be too difficult, provided your capture is started before the persistent connection is started.
Or: if you know the types of services, you could "tell" your script that information and deserialise data that way.
Asked by gvdhoorn on 2021-06-16 06:35:35 UTC
A posteriori analysis is not a problem in principle. But significant disadventage of this approach is, that you collect all TCP traffic data and for my usecase (runtime in hours with many topics at ~10-1000 Hz publishing frenquency) it would generate huge amount of data. I prefer to target the communication. So in the end I used the option with known client and server port and sniff only that communication.
However, if persistent connection breakes, I am in trouble. So it is a compromise...
Asked by m.bahno on 2021-06-16 07:16:06 UTC
It gets a bit academic at this point, but it's possible to use a live-capture and process it live. Possibly even with a small ringbuffer. pyshark
apparently can do this: Reading from a live interface using a ring buffer.
That would avoid capturing all traffic, as you could literally listen in on the XML-RPC traffic and then figure out which packets to keep by filtering for those streams.
(lots of hand waving here of course)
Edit: if you go in this direction I would really be interested in this and possibly contribute.
Asked by gvdhoorn on 2021-06-16 07:24:15 UTC
It gets a bit academic at this point, but it's possible to use a live-capture and process it live.
I suppose it would. Unfortunately, I am forced to go for effective way right now, so as long it works for my use-case, I will stick to that and not to make a complete solution. I am actually using pyshark
right know, but I have some problems with it.
If end up going for the complete solutions, I would like to share it, but I am not sure if I company I work for allows it. But if so, I'll let you know. Personally I also find this as the most complete and elegant solution, but sometimes one has to go for a quick one... Anyway, thanks again for inspiring comments!
Asked by m.bahno on 2021-06-16 08:11:33 UTC
Comments
IIRC, Wireshark has a dissector for ROS traffic built-in for a few years now. I'm not entirely sure whether it supports services though, but that would be easily checked.
Edit: this one: TCP based Robot Operating System protocol (TCPROS).
Asked by gvdhoorn on 2021-06-14 09:40:39 UTC
Thank you for pointing to this option! However, it does not decode ROS services.
Asked by m.bahno on 2021-06-14 10:22:06 UTC
Afaik services use
TCPROS
as well, so I'm curious why it wouldn't dissect it.Edit: testing it a bit, it does seem to be able to dissect the request header, but not the payload. Perhaps it doesn't support or achieve reassembly of the stream properly:
As to whether there is a "simple packet structure": afaik there is no single document which describes that. Perhaps wiki/TCPROS comes closest.
Asked by gvdhoorn on 2021-06-14 10:38:49 UTC
Though this is true only for service connection initialisation (every time for non-persistent services, but only once in the beginning for persistent ones). So for my usecase (starting to listen in the middle of the communication), it does not help. But it is good to know, wireshark provides dissection of some ROS messages.
Asked by m.bahno on 2021-06-15 08:57:00 UTC
So if you don't know the service types involved, how would you go about dissecting the data? Service payload doesn't seem to include any indication of the layout of the data, which seems to make dissection difficult.
Asked by gvdhoorn on 2021-06-15 10:49:15 UTC
I do know the service type. But for persistent TCP service connection, only first packet contains the header and I am not guaranteed to catch it (most of the times I will not). So my goal was to dissect ROS service message WITHOUT header (which persistent services do not send) of known service. Or at least get some service-unique part...
Asked by m.bahno on 2021-06-15 11:17:18 UTC
For me, it is enough to recognise the packet among others (to sniff all packets of one service). Since the service packet is quite long (~ 70 bytes + payload), I supposed there are some bytes telling of what service this packet belongs to (since in one ROS node, more service servers can operate). So far I did simple packet difference and located bytes, which change on different ROS runs for the same service, but that's far from perfect. I also tried to use service URIs (to catch only packets between two ports - client and server), however client port is harder to obtain, can change and this approach is useless for more clients.
Maybe I get a bit off topic in here, but it is a background motivation for my question.
Asked by m.bahno on 2021-06-15 11:18:22 UTC
well that helps. But from what I remember/saw, plain payload of service invocation does not include any indication of which service type is involved. Interpretation of bytes is dependent on external/prior exchanged information between client and server.
Is it really possible to dissect the data without having witnessed the session setup?
is this always the case? The
AddTwoInts
service I tested above has a payload of around 24 bytes I believe (a
,b
and some other number).yes, so that's what I was referring to. If you miss out the header, the rest if just "random" binary data.
Asked by gvdhoorn on 2021-06-15 11:36:30 UTC
The capture above is from running
rosrun roscpp_tutorials add_two_ints_client 1234 5678
(with the server androscore
already running). That's the handshake.The request data is
10000000d2040000000000002e16000000000000
, or:Wireshark shows me the request data in a separate packet. The dissector does not do any reassembly apparently (I'm not sure it could do that, as ROS comm is on random ports, so it'd probably need to witness
roscore
's XML-RPC traffic as well to figure out the ports).I'm interested in this too, so I'm wondering how, without having seen the initial session handshake, you'd interpret this as ROS service payload?
Edit: following the stream does show the complete conversation:
Asked by gvdhoorn on 2021-06-15 11:52:43 UTC
My interpretation was quite naive, I knew values being sent and I looked for them in the raw message :-) . Fortunately, message was short and the payload was in the end, so it went all good.
Ad message size: I think that was my prior misunderstanding, I was talking about the whole TCP message, not just the "ROS" part. So the ROS payload was just a few bytes (see bellow).
From further experiments and looking at the code I found out, that request (now on speaking only about data added by ROS) consists only of 4B of message length + payload (2xfloat32b in my case). Response consists of 1B of reponse status (in my case always 1, see http://wiki.ros.org/ROS/TCPROS) + 4B length + reponse. So I doubt, I can move on from this.
Asked by m.bahno on 2021-06-15 12:20:08 UTC
However I noticed, that you can add so called 'Service connection header'. I am not sure, if this is sent with every message, but if so (and first glance on code suggests it), it may solve the problem, because I can add some unique identifier to it to be recognisable. I will test this tomorrow and share the results.
Asked by m.bahno on 2021-06-15 12:22:15 UTC
This is, I believe, the relevant code creating a service message and it looks to add only header, if provided and then simply serialised message (plus length byte).
Asked by m.bahno on 2021-06-15 12:24:42 UTC
IIRC, the connection header is really only sent during the handshake.
I'm curious to know why you're trying to do this. If possible, could you describe the use-case?
yes, that's the place.
Asked by gvdhoorn on 2021-06-15 12:25:13 UTC
Sure. I need to get service call latency. I experienced some (rare but existing) really high (> 1s) latency service calls during long run (> 5 hours) and I want to find out, if the problem is the network load (TCP packet transport) or elsewhere.
Asked by m.bahno on 2021-06-16 03:33:31 UTC