Your setup is pretty much analagous to setups with GPS + IMU alone, and it's not a use case that is especially well-supported by r_l. If I added IMU bias correction, it might help, but time is an increasingly rare commodity. Another option is that you can try differentiating your poses and sending that velocity to the EKF as well. You're giving it the same info twice, but it might make things a bit smoother overall.