No, you don't have to use a mesh, the primitive should suffice. You need to provide more info for the troubleshooting of the problem. Please also share your srdf file. If the collision between grippers and the table object are allowed, then this kind of a behavior is normal.