intermittent errors with pyserial based service - how to make it robust?

edit

I am working on making a serial communication ROS2 service based on pyserial in order to communicate with my microcontrollers (arduinos which have been left behind by micro-ROS) I would have loved a well supported solution to this but I have not been able to find one so this has been my best attempt. It has generally worked but I am running into an extremely frustrating issue when many requests are put on the service where the serial connection will essentially break for seemingly no reason, returning either of the following (depending on read or write):

    write failed: [Errno 5] Input/output error

or

    device reports readiness to read but returned no data (device disconnected?)

or occasionally the readlines will timeout and appears device is unresponsive

sometimes the device comes up at a new address which is why I have based my service on serial number rather than address, and I have basically programmed in some band aids which attempt to reconnect but while it does seem to reconnect successfully, the connection does not seem to be functional and times out.

I'm seriously stumped, all I really know is that it tends to happen when I am making calls to the service at very high frequency. I have tried increasing the timeouts to over 1 second, all calls to this service use a client.wait_for_service() call before ever sending the request so I don't think there is any reason calls should be overlapping.

well without further ado, here is my serial service. I am happy to answer any questions about it, I am also open to entirely new solutions to communicating with 8-bit microcontrollers

#!/usr/bin/env python3
import rclpy
import serial
import serial.tools.list_ports
import time

from serial import SerialException
from rclpy.node import Node 
from garden_interfaces.srv import SerialMessage


class SendRecieveSerialServerNode(Node):
    def __init__(self):
        super().__init__("send_recieve_serial_server")
        # parameter declcare
        self.declare_parameter("serial_number")
        self.declare_parameter("id_key")
        self.declare_parameter("timeout", .25)

        # get parameter
        self.serial_number = self.get_parameter("serial_number").value
        self.id_key = self.get_parameter("id_key").value
        self.serial_timeout = self.get_parameter("timeout").value

        # create server
        self.server = self.create_service(
            SerialMessage,
            "send_recieve_serial",
            self.callback_send_recieve_serial)

        # get serial port and connect
        self.get_serial_port(self.serial_number)
        self.connect_to_serial(self.serial_port)

    def callback_send_recieve_serial(self, request, response):
        try:
            start = time.process_time()
            # send message and recieve response
            recieved = self.send_and_receive(request.message)
            if recieved is not None:
                recievedKey = recieved[0]
                recievedData = recieved[1]
                # use response key to determine success
                if(recievedKey == request.success_key):
                    response.success = True
                    response.reply_data = recievedData
                    response.response_time = int((time.process_time() - start) * 1000)
                    self.get_logger().info("exchange succeeded in: {}ms".format(response.response_time))
                elif(recievedKey == "err"):
                    response.success = False
                    self.get_logger().error("MCU error: {}".format(recievedData))
                    response.reply_data = recievedData
                else:
                    self.get_logger().error("unknown response key: {}".format(recievedKey))
                    response.success = False

            return response
        except Exception as err:
            self.get_logger().error("exchange failed -> {} attempting reconnect...".format(err))
            self.serial.close()
            self.connect_to_serial(self.get_serial_port(self.serial_number))

    # return matching serial port for given serial number
    def get_serial_port(self, serial_number):
        match_port = None
        all_ports = serial ...