ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question
1

ROS2/pendulum demo (and rttest/example_loop) get killed if 8GB free RAM is not satisfied.

asked 2019-02-06 03:44:13 -0600

Nicolas Gerig gravatar image

updated 2019-03-04 02:28:15 -0600

Hi !

I am using Ubuntu 16.04 with a real-time patched kernel (4.14.93-rt53). I have tested my system with cyclic tests and am happy about the realtime performance.

I have built and installed Ros2 (Crystal Clemmys) according to this manual: https://index.ros.org/doc/ros2/Instal... . The talker and listener examples work fine.

When I try to run the pendulum demo ( https://index.ros.org/doc/ros2/Tutori... ) I get an unexpected output:

$ RMW_IMPLEMENTATION=rmw_connext_cpp pendulum_demo
RTI Data Distribution Service Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User license@rti.com For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
RTI Data Distribution Service Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User license@rti.com For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
Couldn't set scheduling priority and policy: Operation not permitted
Killed

I could get rid of the line:

 Couldn't set scheduling priority and policy: Operation not permitted

I have tried to give the rights according to this: https://index.ros.org/doc/ros2/Tutori...

I can also see (after logon/logoff) that the rights are updated accordingly:

$ ulimit -r
98

Now the output is without this line, but still gets killed:

$ RMW_IMPLEMENTATION=rmw_connext_cpp pendulum_demo
RTI Data Distribution Service Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User license@rti.com For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
RTI Data Distribution Service Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User license@rti.com For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
Killed

I have found a very similar (unanswered) question and initially posted mine (by mistake) as an answer here: https://answers.ros.org/question/3030...

I would be happy to receive any Input. Since I am inexperienced with Ros, it might very well be some rookie mistake. Thank you! Nicolas

Update: Testing with rttest and debugging with gdb

According gvdhoorn's suggestion I have tested example_loop from ros2/realtime_support/rttest. Unfortunately this program gets Killed as well.

To debug the example_loop with gdb I have rebuild and started with

$ cmake ../ -DCMAKE_BUILD_TYPE=Debug 
$ make

$ gdb ./example_loop
(gdb) r

and received the following output:

Starting program: /home/<myUserName>/realtime_support/rttest/examples/build/example_loop
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program ...
(more)
edit retag flag offensive close merge delete

Comments

Can you test whether example_loop from ros2/realtime_support/rttest does work? If that also gets killed, then something is wrong.

You might want to see whether gdb can help clear up the mystery in that case.

gvdhoorn gravatar image gvdhoorn  ( 2019-02-06 04:04:52 -0600 )edit

Btw: your question title still mentions the privilege issue, but in your latest output I don't see it any more.

gvdhoorn gravatar image gvdhoorn  ( 2019-02-06 04:16:02 -0600 )edit

Hi gvdhoorn, Thank you for your support! I have updated the title and tested the example_loop. Unfortunately, the example loop gets killed as well:

./example_loop
Killed

Sorry for taking so long to test your suggestion.

Nicolas Gerig gravatar image Nicolas Gerig  ( 2019-02-07 03:46:53 -0600 )edit

First thing I'd try is run it in gdb.

gvdhoorn gravatar image gvdhoorn  ( 2019-02-07 04:09:50 -0600 )edit

I am trying to do that now (never used gdb before). I have rebuilt example_loop to add debug info: $ cmake ../ -DCMAKE_BUILD_TYPE=Debug $ make Started the debugging by $ gdb ./example_loop It prints infor about gbd, then reads the symbols, and stays with (gdb). I dont see any CPU usage.

Nicolas Gerig gravatar image Nicolas Gerig  ( 2019-02-07 05:05:25 -0600 )edit

Once gdb has started, you type r (short for run) and press enter.

It will probably start outputting all sorts of info. After it gets killed, copy-paste all that into your question (use the edit button/link).

If possible, also type bt and press enter, after it's killed.

gvdhoorn gravatar image gvdhoorn  ( 2019-02-07 05:11:04 -0600 )edit

re: your edit: it could be that your user doesn't have the permission to request the large stacks that realtime_support is trying to use. What is the output of ulimit -a?

Note: I'm not sure here, just guessing at this point, as your program just getting killed isn't very informative.

gvdhoorn gravatar image gvdhoorn  ( 2019-02-07 06:31:46 -0600 )edit

Also: have you tried running the same program on a different machine? How did you create the kernel?

gvdhoorn gravatar image gvdhoorn  ( 2019-02-07 06:33:54 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-03-04 02:28:01 -0600

Nicolas Gerig gravatar image

updated 2019-03-04 02:33:22 -0600

gvdhoorn gravatar image

Found the issue, my bad.

Thank you all for your valuable feedback! I have learnt a lot already while testing the below steps. However, my failure was rather trivial - the PC I have borrowed from our IT did only have 8GB of RAM, which is not enough according to the example tutorial of the pendulum_demo it requires at least 8GB of free RAM (from the Run the tests section):

Before you run make sure you have at least 8Gb of RAM free. With the memory locking, swap will not work anymore.

I failed to check the PC specs and the requirement in the example manual. However, it is interesting to note that not only the pendulum_demo, but also example_loop from ros2/realtime_support/rttest required more RAM than was available in the tested PC. By using htop in a seperate console I could follow how the memory is filled after either programm start and the programm being killed exactly once the memory is full.

edit flag offensive delete link more

Question Tools

1 follower

Stats

Asked: 2019-02-06 03:44:13 -0600

Seen: 842 times

Last updated: Mar 04 '19