# What is the best way to record RGB and Depth data from a Kinect using Openni?

I'm attempting to record short (~10s) video streams from a kinect. Ideally I'd like to output two video files after recording is finished: one file for RGB and the other file for depth (I can save a series of images rather than video file if that's not possible, but that tends to be space intensive, so I'd prefer not to do that). This is so I can process the data later. I have been searching for the past few days around the internet, and while I have found many resources and posts that are close to what I want to do, none of them are quite exactly what I'm looking for. I'm writing in Python, which is relevant since a lot of the posts I've found refer to C++ code, but I can never seem to find the equivalent function calls for Python.

Here's what I've got at the moment, but it is very hacky and doesn't really work. For recording the RGB data, I use image_view extract_images. I make this function call:

subprocess.Popen(["rosrun","image_view","extract_images","_sec_per_frame:=0.01","image:=camera/rgb/image_color"], cwd="temp/rgb/", stdout=DEVNULL)


To record the depth data, I wrote a class that subscribes to the "/camera/depth/image_rect" topic and then writes out the images to files:

class ImageConverter:

def __init__(self):
self.count = 0
self.todo = Queue.Queue()
self.stop = False
self.sub = None

def __getCounter(self):
self.countLock.acquire()
c = self.count
self.count = self.count + 1
self.countLock.release()
ret = str(c)
while len(ret) < 4:
ret = "0" + ret
return ret

def __callback(self, data):
self.todo.put(data)

def __processImage(self,data):
bridge = CvBridge()
try:
source = bridge.imgmsg_to_cv(data, "32FC1")
except CvBridgeError, e:
print e
return

ret = cv.CreateImage((data.width, data.height), 8, 3)

mx = 0.0
for i in range(data.width):
for j in range(data.height):
d = source[j,i]
if d > mx:
mx = d

for i in range(source.width):
for j in range(source.height):
v = source[j,i]/mx*255
ret[j,i] = (v,v,v)

return ret

def startRecording(self,topic):
self.stop = False

def processor():
rate = rospy.Rate(100.0)
while (not self.stop) or (not self.todo.empty()):
if not self.todo.empty():
data = self.todo.get()
img = self.__processImage(data)
c = self.__getCounter()
cv.SaveImage("depth/frame" + c + ".jpg", img)
print("[ImageConverter]: image saved " + c + "/" + str(self.todo.qsize()))
rate.sleep()

t.daemon = True
t.start()

self.sub = rospy.Subscriber(topic, Image, self.__callback)

def stopRecording(self):
self.sub.unregister()
self.stop = True
t.join()


As you can see in the __processImage method, I had to manually convert the image from 32FC1 to bgr8, since everytime I try to call the cv_bridge method imgmsg_to_cv with ...

edit retag close merge delete

Sort by » oldest newest most voted

Ok, so the solution I found is a little specific to my project. I determined that OpenNI .oni files will work as video file output for my purposes. If anyone is reading this and wants to go all the way to .avi files, if you try my solution and then add this:

http://kirilllykov[dot]github[dot]io/blog/2013/03/19/convert-openni-star-dot-oni-files-into-avi/

(replace '[dot]' with '.') solution on top of that, you should be able to get to .avi files.

To get the .oni files from the RGB and depth data, I first downloaded OpenNI (okay, so I already had OpenNI from before, but the ROS version isn't the same as the release version, so I downloaded that and put it in it's own folder) from here:

http://www[dot]openni[dot]org/openni-sdk/

(replace '[dot]' with '.') (Note, I'm running Linux, so this may work slightly differently for you if you are running something else). I extracted the folder and just placed it in my home directory (so it didn't interfere with ROS's OpenNI). Next I downloaded Primesense:

https://pypi[dot]python[dot]org/pypi/primesense/2.2.0.30-3

(replace '[dot]' with '.') which adds python bindings for OpenNI. I ran

setup.py build


and

setup.py install


from the Primesense folder to install the necessary libraries. I then ran the following python code to record the .oni files:

import roslib
import rospy
from primesense import openni2

openni2.initialize(OPENNI_REDIST_DIR)

dev = openni2.Device.open_any()

depth_stream = dev.create_depth_stream()
color_stream = dev.create_color_stream()
depth_stream.start()
color_stream.start()
rec = openni2.Recorder("test.oni")
rec.attach(depth_stream)
rec.attach(color_stream)
print rec.start()

#Do stuff here

rec.stop()
depth_stream.stop()
color_stream.stop()



where OPENNI_REDIST_DIR is the path to the redist folder in the OpenNI library. You can play these back just to double-check that they worked by running 'NiViewer' (in the Tools folder of the OpenNI library) with the path to the .oni file.

more

maybe rosbag

more

What I've done in the past is record with rosbag, and I wrote a utility for generating video files from bags: https://github.com/OSUrobotics/bag2video

( 2014-02-23 03:48:11 -0500 )edit

I've looked at using rosbag, but it seems like it only pushes the problem down the line. For example, this (http://wiki.ros.org/rosbag/Tutorials/Exporting%20image%20and%20video%20data) tutorial explains how to extract video from a rosbag, but why can't I just record straight from the topic rather than storing in a rosbag first? Actually, that tutorial is the reason I made the calls to 'extract_images' and 'mencoder' in the first place, but I didn't see any reason to store in a rosbag and then play back. Dan, thanks for the utility. When I get back into the lab I will see if I can get that to work. If I can, I think I might go with this route.

( 2014-02-23 07:22:59 -0500 )edit

Yeah, I think the real answer is that what you're trying to do is pretty common, but there's no standard solution. It used to be possible with image_saver + mencoder/ffmpeg, but image_saver disappeared.

( 2014-02-23 09:07:23 -0500 )edit

Thanks for your code. I ran it and it works great for the RGB data, but I get the same problem with depth data as in my original post. I get an error thrown at line 40 of bag2video.py: img = np.asarray(bridge.imgmsg_to_cv(msg, 'bgr8')) that OpenCV can't convert from 32FC1 to bgr8. I found another solution which I'll write up, but thanks anyways.

( 2014-02-24 14:31:41 -0500 )edit

Hmm, I had never tried with depth. Even though you found another solution, next time I have a Kinect handy, I'm going to poke around and figure out what the correct conversion is.

( 2014-02-25 03:38:40 -0500 )edit

@rse101, how did you manage to save depth images? I cant see them with image_view (however they are working in RVIZ)

( 2014-03-01 11:47:05 -0500 )edit

Sorry about that. I originally posted my answer a few days ago, but ROS Answers wouldn't post it and it wasn't until today that I figured out that it was because I have links in the answer. I managed to get it posted now, so hopefully that answers your question.

( 2014-03-02 08:42:18 -0500 )edit

@rse101, thanks for your answer. I've just found the video_record tool here: http://wiki.ros.org/image_view . It may work if you alrady can see that topics with image_view.

( 2014-03-04 07:47:39 -0500 )edit