Function call significantly slower in service handler

asked 2016-11-27 14:08:03 -0600

updated 2016-11-27 19:41:03 -0600

I found that if I call my function within a service handler it runs significantly slower (about 10x slower). I have not notice this problem before with my other service calls. I suspect it is because this function call uses a lot of memory, but not knowing how service handlers work under the hood I have no idea how to fix this problem.

My code is copied below. Some insights will be much appreciated. I am currently using ros hydro.

class CNNStateManagerInterface:
def __init__(self, settings):
    self.cnn_state_manager = CNNStateManager(settings)
    s = rospy.Service('get_cnn_list_state', GetListState, self.handle_get_cnn_list_state)
    # function runs normal speed here
    self.cnn_state_manager.get_cnn_list_state(None)

def handle_get_cnn_list_state(self,req):
    # function runs 10 times slower if placed in the handler
    self.cnn_state_manager.get_cnn_list_state(None)

if __name__ == '__main__':
    cnn_state_manager_i = CNNStateManagerInterface(settings)
    rospy.spin()

(Update) Based on gvdhoorn's insight I moved the initialization inside the handler and the function now runs normal speed. However, the initialization takes a lot of time and I am hoping to only need to do it once. Also I am still puzzled by why it slows the whole function down uniformly instead of having an overhead of copying memory to the thread. Appreciated if anyone has a better solution for this.

class CNNStateManagerInterface:
def __init__(self, settings):

    s = rospy.Service('get_cnn_list_state', GetListState, self.handle_get_cnn_list_state)

def handle_get_cnn_list_state(self,req):
    self.cnn_state_manager = CNNStateManager(settings)
    # function runs normal speed now
    self.cnn_state_manager.get_cnn_list_state(None)

if __name__ == '__main__':
    cnn_state_manager_i = CNNStateManagerInterface(settings)
    rospy.spin()

(Update 2) I did some profiling with the python profiling tool yappi. The following are the time differences of a few functions that might be interesting. As you can see, the avg function time and total time between two pycaffe calls are very different.

Initialize in init:

left to right (name    num call    tsub    time total    time avg )

/pycaffe.py:72 Classifier._Net_backward_from_layer    160    0.041332    88.79009    0.554938 tcpros_base.py:650 TCPROSTransport.receive_once    13929    0.564422    14.48369    0.001043

Initialize in handler:

/pycaffe.py:72 Classifier._Net_backward_from_layer    160    0.029367    0.688773    0.004305 tcpros_base.py:650 TCPROSTransport.receive_once    4118    0.115029    1.749794    0.000378

I thought it might be due to using pycaffe functions with multi-threading so I tested threads instead of ROS handler, however the speed is normal when I run it in a thread like the following.

class CNNStateManagerInterface:
    def __init__(self, settings):
        self.cnn_state_manager = CNNStateManager(settings)

        t = threading.Thread(target=self.handle_get_cnn_list_state(None))
        t.start()
    def handle_get_cnn_list_state(self,req):
        # function runs normal speed now
        self.cnn_state_manager.get_cnn_list_state(None)
edit retag flag offensive close merge delete

Comments

I don't know if this is the case here, but I remember some questions about similar things where it turned out that the threads that handle an incoming service request cannot access the state instance of the main thread, so they have to (re)create / copy one of their own. That is rather slow.

gvdhoorn gravatar image gvdhoorn  ( 2016-11-27 14:23:44 -0600 )edit

Thanks gvdhoorn, I think that is closely related. I just moved self.cnn_state_manager = CNNStateManager(settings) into the handler function and the function runs normal speed. What I don't understand is that if it has to create a copy in the thread, it should have an overhead time not a slow down.

Gooly gravatar image Gooly  ( 2016-11-27 15:15:10 -0600 )edit

I'm not sure what is going on, I just happened to remember some other users reporting this. And I believe it might not even be ROS specific (ie: other systems that fork/use a thread pool could be affected). I'd search Google with more generic search terms. Sorry I can't be of more help.

gvdhoorn gravatar image gvdhoorn  ( 2016-11-28 05:06:23 -0600 )edit