How glad I am we're not the only ones seeing this bug! We've just spent about 20+ manhours debugging it. With the help of a network tap device, we pinpointed the issue to the timing of the packets sent by the lidar.

It seems the lidar sends data in bursts (8 UDP fragments, pause 8 UDP fragments, pause, 1 UDP fragment). These 17 fragments form one large fragmented UDP datagram. And that's what the ROS driver sends furter as the lidar packet. When you connect the lidar directly to a computer, you don't observe the packet loss as there is noone else on the cable and a Gbit link is good enough for the lidar. However, when the lidar traffic is mixed in a switch with traffic from other devices, it shows the bursts of data are just too fast and switches start throwing Ouster packets away as their buffers are half-full of the other traffic and can no longer contain all packets belonging to the Ouster bursts. We haven't found any parameter of the switches we tested (about 5 types) that would tell us the size of the buffers, but we assume the store-and-forward technology they use has a buffer for just a few packets. Ouster sends 10000 UDP packets/s in 1024x10 mode, and one of our switches specifies 17 Mpackets/s, which is way more. However, the bursting behavior goes beyond the limit of even this switch on short time intervals.

We plan to report this as an issue to Ouster, as it basically renders the sensor unusable when connected to a switch.

Until it is fixed (if!), there are some things you can do to make things a bit better.

1. Switch to 512x10 mode and connect Ouster to switch via a 4-wire 100 Mbit Ethernet cable. This will force it into 100 Mbit mode, in which the bursts are much smaller. The bandwidth of 512x10 mode is 60 Mbps which is easy to handle on the 100 Mbit link. This option fully resolves the issue for us, but you have to work only with 512x10 mode.

2. Connect the lidar via a dedicated network card (e.g. a USB-Ethernet adapter or a PCIe card). This fully resolves the issue (if you manage to find a reliable USB-Ethernet adapter, which is not easy).

3. If your receiving machine is Linux, set the following in /etc/systctl.d/60-ouster-fix.conf:

net.ipv4.ipfrag_high_thresh = 26000000 net.ipv4.ipfrag_low_thresh = 13000000 net.ipv4.ipfrag_time = 5

You can play with the values. This basically doesn't fix the lost packets problem, but it makes it less serious for the receiving computer. As said earlier, Ouster sends 20k UDP frames fragmented to 17 MTU-sized frames (1500 B). As frames get lost on the switch, the receiving computer adds any received fragment to a buffer where it waits for its other buddies to form the large UDP datagram. However, with 20% of them missing, most datagrams are never completed (reassembled). So their parts just sit in this buffer waiting. The default setting for Ubuntu is to kick them off after 30 seconds. You can see this in action - if you run the lidar and cameras at the same time, every 30 seconds you receive a few damaged scans in quick succession. That's exactly when the reassembly buffer kicks off most of the hopeless datagrams. The other two settings just control size of the reassembly buffer. The default is about 300 kB, we increase it to 26 MB.

With these 3 settings, we got much better performance, but the problem is still not fixed. Without them, traffic of 60 Mbps from other devices is enough for lidar packets to start disappearing, wherease with the settings, most lidar packets make it through even with 500 Mbps other traffic.

I'll update this answer if Ouster tells us some relevant information or if they fix the issue in a new firmware.