How do you convert non-velodyne pcap file to bag file.

asked 2020-12-06 18:03:15 -0500

AdamS gravatar image

I am working with a pcap file from a non-velodyne sensor. Is there a generic way to convert non-velodyne pcap files to bag files?

The goal is to play back this file and use the data from the topics. I will likely need to build a rosnode for this sensor and a publisher for the information regardless. However right now I only have the pcap file. I was told by the manufacturer that there should be a ros endpoints in the file.

I am wanting to convert the .pcap to .bag or simply play the .pcap file so that i can poke and prod it with the ros tools available to me. However the only explanation on how to do this is for Velodyne. Wich throws errors when I attempt to convert the pcap file. ([ WARN] [1607297984.564623989]: Error -2 reading Velodyne packet:). The obvious reason for this error is that the velodyne drivers are incompatable with the sensor file I have. but I figured it would try it anyways.(one can hope)

i am curious if there is a way to do this. any help or advise you can offer is welcome.

Technical Details: I am on ROS Melodic on ubuntu 18.04 unfortunately, im under NDA and cant reveal the company name or model I am working on nor can I provide the file.

edit retag flag offensive close merge delete


Just thinking about this: a .pcap file contains a recording of (raw) network traffic. A .bag contains serialised ROS messages. The former is essentially (almost) a verbatim dump of what was received and transmitted over a specific NIC. The latter is an ordered and 'optimised' for replay recording of message exchange at the (OSI) application layer.

What would a "generic conversion from .pcap to .bag" look like? Unless there is actual ROS message traffic in a .pcap file, there is not really any specific conversion from captured packets to ROS messages possible: at which level should such a conversion be done? Individual UDP/TCP frames? Combined TCP frames? Raw ethernet packets? And at which level? Network, transport, session, application?

Conversion from .pcap to .bag can be done, but it will likely always be a conversion embedded in a context, as otherwise it's impossible to convert in a way which ...(more)

gvdhoorn gravatar image gvdhoorn  ( 2020-12-07 01:00:26 -0500 )edit

I guess what I'm trying to say is: I don't believe there will be a really useful generic way of converting a .pcap to a .bag, as without knowing what kind of traffic the .pcap contains (ie: context), it will be almost impossible to interpret the raw traffic into ROS messages.

The .pcap converter for Velodyne captures is exactly one of those converters which exploits context: it searches the file you pass it for "Velodyne packets", and knows how to convert those to ROS messages. But it doesn't know "anything else" as you found, as passing it just "any" .pcap results in errors.

Unless someone has already written a converter for network traffic of your specific device to ROS messages, I don't believe you'll be able to process the .pcap you have.

gvdhoorn gravatar image gvdhoorn  ( 2020-12-07 01:04:48 -0500 )edit

I was told by the manufacturer that there should be a ros endpoints in the file.

I'm not sure what this means. If it means there is actual ROS traffic in the file (ie: the .pcap contains a capture of the raw data exchanged by ROS nodes), you could write a converter for that.

But that would again be a converter which exploits context: you'd have embedded the knowledge that there is ROS traffic on certain ports, which would make it possible to convert that into serialised structures in a .bag.

Edit: distributing .pcaps of ROS traffic seems like a strange thing to do to me, but perhaps your OEM has its reasons.

gvdhoorn gravatar image gvdhoorn  ( 2020-12-07 01:08:02 -0500 )edit

@gvdhoorn once again you have some awesome insights and have helped me out. I think the best approach is to use something like tcpreplay to replay the pcap file as if it was network traffic and write a publisher which translates the UDP traffic to a rostopic that can then be read and recorded.(after some data sleuthing of course)

This would likely need to be converted at the transport layer given the limited information about what is on the file. I might have misinterpreted what they were saying about the ros data being on the file as well.

I will update this thread with a solution(which might be multi part) when I build one. this might take me a week or so.

AdamS gravatar image AdamS  ( 2020-12-07 01:35:08 -0500 )edit

What you could do is something like this:

  1. write a deserialiser for the traffic coming out of your device (something you'll have to do quite likely anyway, as it seems you're going to be writing a driver)
  2. create a proper message (or set of messages) to which this device-specific data can be converted in a meaningful way
  3. see if you can reuse the Velodyne driver's infrastructure for reading and iterating over the .pcap contents, but replace all Velodyne specific stuff with your own (from steps 1 and 2)
  4. instead of replaying the .pcap, use your newly created functionality to directly iterate over the frames (datagrams?), convert to ROS messages -> publish

An alternative for the last bullet: instead of publishing, use rosbag's C++ API and write out the .bag directly from your converter process. That would likely be faster and more deterministic.

If I were doing ...(more)

gvdhoorn gravatar image gvdhoorn  ( 2020-12-07 02:51:07 -0500 )edit