Why is message processing in sporadic bursts ?
I am trying to increase the responsiveness of a control loop. I am controlling the pan/tilt servos of a camera (attached to a Raspberry Pi 3) based on the center of a bounding box found by YOLO vision system (running on a desktop pc with GTX 1080 ti GPU). I am observing very jumpy servo movements.
So:
Yolo(on pc) <----wifi------ PiCam(Pi3)
PID(on pc) -----wifi-----> ServoController(Pi3)
The desired servo positions are sent as messages by the PID controller on the pc at 10 Hz (this is confirmed by the messages seen on rqt_console and rostopic Hz). The message queue size of the publisher is 1.
On the Pi3 I have set the loop rate of the servo controller to values between 10 and 50 Hz. I have set the subscriber queue size to 1 and 10.
- With a ServoController queue size of 1, rqt_console shows only about one in 10 of the control messages are processed. A control signal sent to the servos at 1 or 2 Hz resulting in very jumpy movements.
- With a ServoController queue size of 10, rqt_console shows that all the control messages are processed, but they are processed very quickly in blocks of about 10, then nothing happens for almost a second, then another block of 10 are processed. The servos remain very jumpy because all the small incremental movements are processed at the same time and the largest step dominates.
My conclusion is that the controller on the Pi3 is processing in bursts, but why ? Changing the loop rate from 10 to 50 did not improve matters. I checked the cpu utilisation on the Pi3 but it is never more than 7%.
Here is output from rqt_console which shows the bursts. 'I heard' denotes a message processed by the servo controller. PID denotes a message sent from the PID loop. ```
"I heard: [14, 99]",/PiServoNode,1559747410.29711
"I heard: [14, 101]",/PiServoNode,1559747410.29413
"I heard: [14, 124]",/PiServoNode,1559747410.29118
"I heard: [14, 113]",/PiServoNode,1559747410.2879
"PID X: pan=99, err=84, cumErr=204, P=92.400000 I=12.240000, clamp=10000
",/eye_control_node,1559747409.25256
"PID X: pan=101, err=91, cumErr=120, P=100.100000 I=7.200000, clamp=10000
",/eye_control_node,1559747409.15253
"PID X: pan=124, err=118, cumErr=29, P=129.800000 I=1.740000, clamp=10000
",/eye_control_node,1559747409.05254
"PID X: pan=113, err=114, cumErr=-89, P=125.400000 I=-5.340000, clamp=10000
",/eye_control_node,1559747408.95256
"I heard: [14, 101]",/PiServoNode,1559747409.1169
"I heard: [14, 91]",/PiServoNode,1559747409.11382
"I heard: [14, 78]",/PiServoNode,1559747409.11077
"I heard: [14, 65]",/PiServoNode,1559747409.10774
"I heard: [14, 54]",/PiServoNode,1559747409.10459
"I heard: [14, 43]",/PiServoNode,1559747409.1008
"I heard: [14, 31]",/PiServoNode,1559747409.09732
"I heard: [14, 20]",/PiServoNode,1559747409.09406
"I heard: [14, 13]",/PiServoNode,1559747409.09102
"I heard: [14, 18]",/PiServoNode,1559747409.0879
"PID X: pan=101, err=109, cumErr=-203, P=119.900000 I=-12.180000, clamp ...
I believe there are many older Q&As already discussing this. See #q303464, #q237592, #q272375 and #q290122. Did you not find those when searching for "messages bursts"?
The first thing to check is whether this is due to Nagle.
No I did not find those, but my original searches (and the original title of my post) used the word 'chunks' instead of 'bursts'. I edited it later when I thought of a better word ! Thanks for a great pointer, I had never come across any of that material in the ROS tutorials that I have read.