As this question contains a singularity related issue it was discussed on the github singularity issue page. The solution I created, based on comments made by @gvdhoorn and @jmstover, can be found below.

Solution

Unfortunately as the singularity %runscript uses the /bin/sh you need a separate script to source a setup.bash file. I, therefore, in the runscript now open a /bin/bash shell using the -rcfile argument. Following in this rcfile called .singularity_bashrc I source the setup.bash files. In singularity versions >3.2.x you now also have the ability to directly do this from within the runscript by using a shebang line.

Steps

1). Clone your .bashrc_file inside your singularity container.

### Clone .singularity_bashrc repository file ##
bash -c "cd / \
    && git clone https://github.com/rickstaa/pandasim_singularity_recipes \
    && cp /pandasim_singularity_recipes/.singularity_bashrc . \
    && rm -rf pandasim_singularity_recipes"

Alternatively, you can also add the following code to the %post section of your singularity recipe to create the .bashrc file from within your recipe file.

## Add extra commands to singularity entrypoint script
touch /.singularity_bashrc.sh
chmod +x /singularity_bashrc.sh
cat <<-EOF >> /singularity_bashrc.sh
    #!/bin/bash
    set -e

    # setup ros environment
    source "/opt/ros/\$ROS_DISTRO/setup.bash"
    echo "Ros setup bash file sourced."

    # Setup Panda_simulation enviroment (chained to franka_ros and moveit)
    if [ -f "/pandasim_ws/devel/setup.bash" ]; then
        source "/pandasim_ws/devel/setup.bash"
        echo "Panda_simulation setup bash file sourced."
    fi
EOF

!!!!Use tabs instead of 4 spaces. More information on heredoc list indenting can be found here !!!!

2). Add the following code in the %runscript section:

## Run custom entrypoint BASH script
OCI_ENTRYPOINT='"/custom_entrypoint.sh"'
OCI_CMD='"/bin/bash"'
SINGULARITY_OCI_RUN="${OCI_ENTRYPOINT} ${OCI_CMD} '-rcfile' '/.singularity_bashrc'"
eval ${SINGULARITY_OCI_RUN}

3). Build the singularity container using the following build command:

sudo singularity build --sandbox singularity_container singularity_recipe.def

4). Now you can call the singularity container using the singularity run command:

singularity run --nv singularity_container