Wrong (extremely non-optimal) MoveIt trajectories on scene with collision objects

edit

First of all RRT is well-known to produce non-optimal paths and especially to randomly produce off-the-wall trajectories in some cases (the second "R" stands for "random" after all). Path simplification often ends up with a very nice trajectory, but it also does not avoid the problem 100% of the time.

Secondly, and I believe this is more likely the issue you observe, you rely on pose targets which might yield multiple IK solutions. If you rely on pose targets and not on specific joint goals, the random exploration is free to prefer other IK solutions. So while you might think you are asking "to move up a little", you are indeed asking to "move to this arbitrary location". If the first IK solution that can be connected is not the closest one (RRT is rapidly-exploring so this is not unlikely to happen), even the shortest trajectory to go there might look awkward.

Gijs already asked about the information on the configuration of your IK solver and solve_type: Distance is the correct choice to reduce the problem. Nonetheless, I would expect you still have problems with this.

kinematics is almost identical to UR robot

It might help to reduce the joint ranges of your robot to match specific use case. If there is only a single IK solution, planning can't succeed to move to an unintuitive solution instead.

Alternatively you might want to prefer Jacobian approaches as implemented in moveit_servo.

EDIT:

Another alternative to improve paths that are simply not smooth enough is to use the relatively new chomp optimizer adapter that can optimize a trajectory after the planner finished. This will not magically "unknot" convoluted trajectories, but it can improve RRT-generated trajectories.

To add to @v4hn's answer, a common workaround to avoid this issue is to use joint goals or named poses (you can set them up in the setup assistant (roslaunch your_moveit_config setup_assistant.launch)) when you want to move to a distant goal. That ensures that your robot is in a known configuration, and for nearby pose goals the IK will generally behave well. But you should still make sure the configuration does not flip, e.g. by limiting the joint ranges of your robot as suggested in the other answer.