Robotics StackExchange | Archived questions

[ROS2][RViz2] can't compare times with different time sources

Hi!

When I try to use frame_locked = True in my marker, I am getting the following error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  can't compare times with different time sources

In my application there is only one node and timestamp is obtained with node.get_clock().now(). I found out that this error does not happen if I disable visualization of the marker in question in RViz. Any idea on how to troubleshoot this?

EDIT: Here is an example script that demonstrates the problem. The error message appears as soon as one attempts to add mesh_test topic in RViz menu.

#!/usr/bin/env python3
import os
import sys
import subprocess
import time
import rclpy
from rclpy.qos import QoSHistoryPolicy, QoSProfile
from tf2_msgs.msg import TFMessage
from geometry_msgs.msg import TransformStamped
from visualization_msgs.msg import Marker, MarkerArray
from geometry_msgs.msg import Point, Vector3
from std_msgs.msg import ColorRGBA

MESH_POSITION = Point(x=-1.07,
                      y=0.0,
                      z=0.0)

def bring_up_rviz():
    rviz = subprocess.Popen(
        'ros2 run rviz2 rviz2',
        shell=True)
    def rvizkiller():
        return rviz.poll() is not None
    return rvizkiller

if __name__ == "__main__":

    rclpy.init(args=sys.argv)
    node = rclpy.create_node('SFViewer')
    qos_profile = QoSProfile(depth=1)
    qos_profile.history = QoSHistoryPolicy.RMW_QOS_POLICY_HISTORY_KEEP_LAST

    tfpublisher = node.create_publisher(TFMessage, '/tf', qos_profile=qos_profile)
    meshpublisher = node.create_publisher(MarkerArray, 'mesh_test', qos_profile=qos_profile)

    should_exit = bring_up_rviz()

    x = 0.
    y = 0.
    while rclpy.ok() and not should_exit():
        ros_time = node.get_clock().now()

        x = x+1
        y = y+1

        tf = TransformStamped()
        tf.header.frame_id = 'map'
        tf.child_frame_id = 'test'
        tf.transform.translation.x = x
        tf.transform.translation.y = y
        tf.transform.translation.z = 0.
        tf.transform.rotation.w = 1.
        tf.transform.rotation.x = 0.
        tf.transform.rotation.y = 0.
        tf.transform.rotation.z = 0.
        tf.header.stamp = ros_time.to_msg()
        tfpublisher.publish(TFMessage(transforms=[tf]))


        output = MarkerArray()
        marker = Marker()
        marker.id = 1
        marker.frame_locked = True
        marker.action = Marker.ADD
        marker.header.frame_id = 'test' 
        marker.header.stamp = ros_time.to_msg()
        marker.type = Marker.SPHERE
        marker.mesh_use_embedded_materials = False
        marker.scale = Vector3(x=10.0, y=10.0, z=10.0)
        marker.color = ColorRGBA(r=0.5, g=0., b=0.5, a=0.8)
        marker.pose.position = MESH_POSITION
        marker.ns = 'Position'
        output.markers.append(marker)
        meshpublisher.publish(output)

        time.sleep(1.0)

Asked by xbaelus on 2019-11-18 07:17:40 UTC

Comments

This question https://answers.ros.org/question/303023/runtime-error-from-rclcpptime/ is similar to yours. One of the answers suggest to have a look here http://design.ros2.org/articles/clock_and_time.html . Basically, you are allowed to use different clocks in ROS2 (system clock, ros clock, simulated time), but you are only allowed to compare times coming from the same clock. Apparently part of your application is using a different clock (presumably the visualization marker). You can check this by using

ros2 topic echo /<topic_name>

And seeing if the stamp is different from the stamp of other topics/messages. The fix would be to ensure everything is done using the same clock. I don't have much experience with this so I can't really help you more than this, I just hope it gives you some starting point for troubleshooting.

Asked by MCornelis on 2019-11-18 11:47:01 UTC

As the error states you cannot compare times from different time sources. It doesn't make sense to compare simulated time to wall clock time, they have different timelines. (System time starts at the Unix Epoch some 1.5 billion seconds, versus simulation time usually starts at 0 at startup.)

This error is coming because different parts of your system are using different time sources. To give you anything more specific you'll need to share more about how to reproduce your issue. Preferably in a small example program to isolate the issue.

Asked by tfoote on 2019-11-18 13:51:38 UTC

MCornelis: I have seen this question before, it doesn't have a solution. As I mentioned, in my application I have only one ROS node and one way to get the time. ros2 topic echo is not providing any information about clock source used in header stamp.

Asked by xbaelus on 2019-11-19 06:04:13 UTC

tfoote again as I mentioned, I only use one node and one method to get the time, is it possible that clock source changes without my knowledge? I'll try to prepare a simple example if it's possible :)

Asked by xbaelus on 2019-11-19 06:08:17 UTC

tfoote I have now updated my question with example script that demonstrates the problem.

Asked by xbaelus on 2019-11-19 07:49:59 UTC

My point with ros2 topic echo was to figure out if all topics are using the same clock by inspecting the actual values of the stamps. If 1 topic is at 100 seconds and another is at 1.5 billion then this would indicate one of them has to change. Agreed this does not solve your problem, but it does indicate where the problem may be.

According to your question the marker that gets its stamp from node.get_clock().now() gives an error, so either pass arguments to other parts of your application to make them use the same stamp as node.get_clock().now() or replace the part of the code where you fill the stamp with something that complies with the other part of the application (look online for other ways to generate a timestamp that is compatible).

Asked by MCornelis on 2019-11-19 07:59:27 UTC

Maybe I wasn't clear enough, the ONLY way to obtain the timestamp in my application is node.get_clock().now(). See my updated question with example source code.

Asked by xbaelus on 2019-11-19 08:03:42 UTC

Answers

Thanks for the simple example. The reason you can't find the issue in your code is that what's crashing is not your code but rviz itself. With using the subprocess to launch it you're intermingling the console outputs which makes it hard to know where the exception is coming from.

In general I'd recommend using launch for bringing up multiple processes as it will be a little easier to see what process died and you keep your logic and configuration management separate.

However this looks like a bug in rviz2 or the markers array plugin as publishing any marker should not crash rviz. Please open a ticket at https://github.com/ros2/rviz/issues/new and link to here from there, as well as comment back here with the link to the issues so people can find the other thread both ways.

Note that when I was debugging I redirected the rviz output to file to isolate it using this changed function:

def bring_up_rviz():
    outfile = open('/tmp/mesh_test_out.txt', 'w')
    rviz = subprocess.Popen(
        'ros2 run rviz2 rviz2',
        shell=True, stdout=outfile, stderr=subprocess.STDOUT)
    def rvizkiller():
        return rviz.poll() is not None
    return rvizkiller

Asked by tfoote on 2019-11-19 17:44:41 UTC

Comments

Thanks for help with troubleshooting :) I have created bug report now. I have a question, when you mentioned using launch instead of subprocess, what exactly do you mean (sorry, new to ROS)? I used subprocess to be able to monitor if RViz is still running, if i.e. user closes RViz window I want to stop entire application. Is there a better way to do this?

Asked by xbaelus on 2019-11-19 18:25:04 UTC

ROS in designed to be a distributed system and as such we have tools for launching multiple nodes. You can read more about it here: https://index.ros.org/doc/ros2/Tutorials/Launch-system/

You can setup triggers to shutdown in the case that a specific process stops, or set of processes. As well you can trigger restarts and much more. And there's markup language so you can compose things quickly and reuse components instead of hard coding the connected behavior.

Asked by tfoote on 2019-11-19 18:40:40 UTC