Ask Your Question
1

What is the best way to record RGB and Depth data from a Kinect using Openni?

asked 2014-02-22 16:04:10 -0600

rse101 gravatar image

updated 2016-10-24 09:10:54 -0600

ngrennan gravatar image

I'm attempting to record short (~10s) video streams from a kinect. Ideally I'd like to output two video files after recording is finished: one file for RGB and the other file for depth (I can save a series of images rather than video file if that's not possible, but that tends to be space intensive, so I'd prefer not to do that). This is so I can process the data later. I have been searching for the past few days around the internet, and while I have found many resources and posts that are close to what I want to do, none of them are quite exactly what I'm looking for. I'm writing in Python, which is relevant since a lot of the posts I've found refer to C++ code, but I can never seem to find the equivalent function calls for Python.

Here's what I've got at the moment, but it is very hacky and doesn't really work. For recording the RGB data, I use image_view extract_images. I make this function call:

subprocess.Popen(["rosrun","image_view","extract_images","_sec_per_frame:=0.01","image:=camera/rgb/image_color"], cwd="temp/rgb/", stdout=DEVNULL)

To record the depth data, I wrote a class that subscribes to the "/camera/depth/image_rect" topic and then writes out the images to files:

class ImageConverter:

NUM_THREADS = 1

def __init__(self):
    self.count = 0
    self.todo = Queue.Queue()
    self.stop = False
    self.sub = None
    self.threads = None
    self.countLock = threading.Lock()

def __getCounter(self):
    self.countLock.acquire()
    c = self.count
    self.count = self.count + 1
    self.countLock.release()
    ret = str(c)
    while len(ret) < 4:
        ret = "0" + ret
    return ret

def __callback(self, data):
    self.todo.put(data)

def __processImage(self,data):
    bridge = CvBridge()
    try:
        source = bridge.imgmsg_to_cv(data, "32FC1")
    except CvBridgeError, e:
        print e
        return

    ret = cv.CreateImage((data.width, data.height), 8, 3)

    mx = 0.0
    for i in range(data.width):
        for j in range(data.height):
            d = source[j,i]
            if d > mx:
                mx = d

    for i in range(source.width):
        for j in range(source.height):
            v = source[j,i]/mx*255
            ret[j,i] = (v,v,v)

    return ret

def startRecording(self,topic):
    self.stop = False

    def processor():
        rate = rospy.Rate(100.0)
        while (not self.stop) or (not self.todo.empty()):
            if not self.todo.empty():
                data = self.todo.get()
                img = self.__processImage(data)
                c = self.__getCounter()
                cv.SaveImage("depth/frame" + c + ".jpg", img)
                print("[ImageConverter]: image saved " + c + "/" + str(self.todo.qsize()))
            rate.sleep()

    self.threads = []
    for i in range(ImageConverter.NUM_THREADS):
        t = threading.Thread(target=processor, args = ())
        t.daemon = True
        t.start()
        self.threads.append(t)

    self.sub = rospy.Subscriber(topic, Image, self.__callback)

def stopRecording(self):
    self.sub.unregister()
    self.stop = True
    for t in self.threads:
        t.join()

As you can see in the __processImage method, I had to manually convert the image from 32FC1 to bgr8, since everytime I try to call the cv_bridge method imgmsg_to_cv with ... (more)

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
0

answered 2014-03-02 08:38:34 -0600

rse101 gravatar image

updated 2014-03-02 08:40:52 -0600

Ok, so the solution I found is a little specific to my project. I determined that OpenNI .oni files will work as video file output for my purposes. If anyone is reading this and wants to go all the way to .avi files, if you try my solution and then add this:

http://kirilllykov[dot]github[dot]io/blog/2013/03/19/convert-openni-star-dot-oni-files-into-avi/

(replace '[dot]' with '.') solution on top of that, you should be able to get to .avi files.

To get the .oni files from the RGB and depth data, I first downloaded OpenNI (okay, so I already had OpenNI from before, but the ROS version isn't the same as the release version, so I downloaded that and put it in it's own folder) from here:

http://www[dot]openni[dot]org/openni-sdk/

(replace '[dot]' with '.') (Note, I'm running Linux, so this may work slightly differently for you if you are running something else). I extracted the folder and just placed it in my home directory (so it didn't interfere with ROS's OpenNI). Next I downloaded Primesense:

https://pypi[dot]python[dot]org/pypi/primesense/2.2.0.30-3

(replace '[dot]' with '.') which adds python bindings for OpenNI. I ran

setup.py build

and

setup.py install

from the Primesense folder to install the necessary libraries. I then ran the following python code to record the .oni files:

import roslib
import rospy
from primesense import openni2

openni2.initialize(OPENNI_REDIST_DIR)    

dev = openni2.Device.open_any()

depth_stream = dev.create_depth_stream()
color_stream = dev.create_color_stream()
depth_stream.start()
color_stream.start()
rec = openni2.Recorder("test.oni")
rec.attach(depth_stream)
rec.attach(color_stream)
print rec.start()

#Do stuff here

rec.stop()
depth_stream.stop()
color_stream.stop()

openni2.unload()

where OPENNI_REDIST_DIR is the path to the redist folder in the OpenNI library. You can play these back just to double-check that they worked by running 'NiViewer' (in the Tools folder of the OpenNI library) with the path to the .oni file.

edit flag offensive delete link more
3

answered 2014-02-22 20:18:31 -0600

hamid gravatar image

updated 2014-02-22 20:20:41 -0600

maybe rosbag

edit flag offensive delete link more

Comments

What I've done in the past is record with rosbag, and I wrote a utility for generating video files from bags: https://github.com/OSUrobotics/bag2video

Dan Lazewatsky gravatar imageDan Lazewatsky ( 2014-02-23 03:48:11 -0600 )edit

I've looked at using rosbag, but it seems like it only pushes the problem down the line. For example, this (http://wiki.ros.org/rosbag/Tutorials/Exporting%20image%20and%20video%20data) tutorial explains how to extract video from a rosbag, but why can't I just record straight from the topic rather than storing in a rosbag first? Actually, that tutorial is the reason I made the calls to 'extract_images' and 'mencoder' in the first place, but I didn't see any reason to store in a rosbag and then play back. Dan, thanks for the utility. When I get back into the lab I will see if I can get that to work. If I can, I think I might go with this route.

rse101 gravatar imagerse101 ( 2014-02-23 07:22:59 -0600 )edit

Yeah, I think the real answer is that what you're trying to do is pretty common, but there's no standard solution. It used to be possible with image_saver + mencoder/ffmpeg, but image_saver disappeared.

Dan Lazewatsky gravatar imageDan Lazewatsky ( 2014-02-23 09:07:23 -0600 )edit

Thanks for your code. I ran it and it works great for the RGB data, but I get the same problem with depth data as in my original post. I get an error thrown at line 40 of bag2video.py: img = np.asarray(bridge.imgmsg_to_cv(msg, 'bgr8')) that OpenCV can't convert from 32FC1 to bgr8. I found another solution which I'll write up, but thanks anyways.

rse101 gravatar imagerse101 ( 2014-02-24 14:31:41 -0600 )edit

Hmm, I had never tried with depth. Even though you found another solution, next time I have a Kinect handy, I'm going to poke around and figure out what the correct conversion is.

Dan Lazewatsky gravatar imageDan Lazewatsky ( 2014-02-25 03:38:40 -0600 )edit

@rse101, how did you manage to save depth images? I cant see them with image_view (however they are working in RVIZ)

aguadopd gravatar imageaguadopd ( 2014-03-01 11:47:05 -0600 )edit

Sorry about that. I originally posted my answer a few days ago, but ROS Answers wouldn't post it and it wasn't until today that I figured out that it was because I have links in the answer. I managed to get it posted now, so hopefully that answers your question.

rse101 gravatar imagerse101 ( 2014-03-02 08:42:18 -0600 )edit

@rse101, thanks for your answer. I've just found the video_record tool here: http://wiki.ros.org/image_view . It may work if you alrady can see that topics with image_view.

aguadopd gravatar imageaguadopd ( 2014-03-04 07:47:39 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

3 followers

Stats

Asked: 2014-02-22 16:04:10 -0600

Seen: 9,666 times

Last updated: Mar 02 '14