fix(api): Increase Pyro5 daemon thread pool to account for maintenance runs and multiple modules

[CaseyBatten]

Overview

Covers RQA-5576, RQA-5603, RQA-5608, RQA-5594

When a Flex is setup with lots of modules, a lot more requests are made and handled by each of the processes. On top of this, maintenance runs are run as part of the robot server, not as part of a separate process, so the robot server inherits all the pyro outbound and inbound requests that would have been under a separate process before. This means our original threadpool overhead of 100 for the daemon server was not enough.

Some testing was done on runs before and after this change. Throughout different situations (calibration flows, normal runs) a fully decked-out Flex (Modules in every slot) was seeing a range from as low as 20 to as high as 150 sockets opening and closing on the robot server process at different times. The hardware api process was seeing a range from 14 to 105 during spikes. The threadpool was doubled to 200 (meaning each process can handle up to 200 inbound requests at the same time) to give us a reasonable overhead to ensure we can continue to operate, even under spike scenarios.

Test Plan and Hands on Testing

Calibration tests (on a fully decked out Flex)

• Gripper calibration
• Pipette calibration
• Temp deck calibration
• Heater shaker calibration
• Thermocycler calibration
• Absorbance reader calibration
• Flex stacker calibration

Protocol run tests:

• Run the full module concurrency protocol and ensure it completes without a no free workers threadpool error

Changelog

Bumped the threadpool maximum for the pyro daemon of each process up to 200.

Review requests

Risk assessment

Low

[sfoster1]

this makes sense to me, nice!