fastdss discovery-server 50% operational in docker

edit

Give it a try to set Docker IPC mode to host:

version: "3.3"
services:
  discovery-server:
    image: ros:humble
    environment:
      RMW_IMPLEMENTATION: rmw_fastrtps_cpp
    network_mode: host
    ipc: host
    command: /bin/bash -c 'source /opt/ros/humble/setup.bash && fastdds discovery -i 0'
  talker:
    image: ros:humble
    environment:
      RMW_IMPLEMENTATION: rmw_fastrtps_cpp
      ROS_DISCOVERY_SERVER: 127.0.0.1:11811
    network_mode: host
    ipc: host
    command: /bin/bash -c 'apt-get update -qq && apt-get install -qqy ros-humble-demo-nodes-cpp && source /opt/ros/humble/setup.bash && ros2 run demo_nodes_cpp talker'
  listener:
    image: ros:humble
    environment:
      RMW_IMPLEMENTATION: rmw_fastrtps_cpp
      ROS_DISCOVERY_SERVER: 127.0.0.1:11811
    network_mode: host
    ipc: host
    command: /bin/bash -c 'apt-get update -qq && apt-get install -qqy ros-humble-demo-nodes-cpp && source /opt/ros/humble/setup.bash && ros2 run demo_nodes_cpp listener'

TL;DR

You are running into a limitation of the default SHM transport. To try to identify whether two nodes are running in the same host machine, Fast DDS relies on the available network interfaces. Since your compose sets the network_mode to host, all three container instances will have the exact same network interfaces, and therefore Fast DDS will think that your three nodes are running on the same host, so it will default to the SHM transport. However, since you have container isolation, the nodes do not share the same shared memory segments, thus communication fails. The easiest for you would be to set the IPC mode to host as well (I have verified that this solves the problem locally).