Ask Your Question
0

Conversion of ROS Bag to CSV Extremely Slow

asked 2014-10-21 16:01:53 -0500

bjem85 gravatar image

Hi All,

I have gathered data using rosbag, but listing topics from it and extracting data from it is extremely slow. It takes over an hour to process a .bag file that is 180 MB in size and contains 47 data streams, with output file sizes ranging from 330 bytes to 23 MB. It's operating principle is similar to the answer by user x75 in a closely related question.

Is there anything I can do to improve the parsing speed? My guess is that it is going through the file in full each time, whereas in reality I could save and reuse information from the .bag file in the first pass so I don't have to re-read the whole file every time I extract topic data from it.

#!/usr/bin/env bash

bagfile="${1##*/}"
bagfile="${bagfile%.*}"

# Create the output directory.
mkdir -p $bagfile

topic_list_orig=`rostopic list -b $1`
topic_list=$(echo "$topic_list_orig" | sort)

for topic in $topic_list
do 
    outfname="$bagfile/${topic//\//_}.csv"
    echo "Writing CSV data from topic" $topic "to file" $outfname

    # Check if file exists. If file exists, do not write to it.
    if [ -e $outfname ]
    then
        echo "File" $outfname "already exists. Taking no action"
    else
        # Create a temporary file
        tmpfile=`mktemp`
        echo $tmpfile
        rostopic echo -p -b $1 $topic > $tmpfile

        # Copy temp file into the final file.
        mv $tmpfile $outfname
    fi
done

##### END OF FILE #####
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
2

answered 2014-10-21 16:21:34 -0500

ahendrix gravatar image

You are correct; it is reading through the bag file once for each topic.

Take a look at the rosbag API and the rosbag cookbook for some examples of better ways to extract data from bag files.

You may be able to use the same message to CSV conversion that rostopic uses if you're clever about it.

edit flag offensive delete link more

Comments

I had a look at the examples but they don't cover the case of converting to CSV directly. I know there is a Python API for writing CSV files but it would be good to know how it works with the rosbag API.

bjem85 gravatar image bjem85  ( 2014-10-23 04:25:46 -0500 )edit

It may be useful to read the portion of rospy which converts messages to CSV format: https://github.com/ros/ros_comm/blob/...

ahendrix gravatar image ahendrix  ( 2014-10-23 14:25:47 -0500 )edit

It may be more productive to instantiate a CallbackEcho object and use that, rather than calling the private functions of rostopic directly.

ahendrix gravatar image ahendrix  ( 2014-10-23 14:26:41 -0500 )edit

I will have a look at that. Just note that I will probably need to process about 10 more sets of data, at 1-1.5 hours per piece, or less if I am more selective about the topics I extract. It seems like the time taken to have to learn the new API will not be justified in my case. Am I correct?

bjem85 gravatar image bjem85  ( 2014-10-27 00:01:01 -0500 )edit

If you know Python, it'll probably take 2-4 hours to write the appropriate script. You may also want to try the bag2csv.py answer from the question that you linked to. I wouldn't be suggesting this approach if I thought it was a waste of your time.

ahendrix gravatar image ahendrix  ( 2014-10-27 11:35:17 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2014-10-21 16:01:53 -0500

Seen: 1,015 times

Last updated: Oct 21 '14