Communicator refactor

[bknueven]

This PR finishes the work started in #476 by utilizing the new API introduced for communication in #476 at a higher level. This refactoring has a few major outcomes:

1. The "Hub" is now primarily responsible for (a) termination and (b) sending and receiving the most up-to-date inner and outer bound information (c) keeps track of the spoke having the best bounds.
2. Spokes must now explicitly check for updates in the fields they are interested in: the call to Spoke.got_kill_signal only updates the SHUTDOWN field. This mirrors how things are currently done in the hub and removes from spooky action at a distance from the code.
3. With the exception of termination and best current bounds (which are still hard coded to rank 0), Spoke now look up the rank providing the information they need to function. If there is no provider of that information the program will terminate with an error, and if there is more than one provider of that information the program will typically terminate with an error.
4. Lots of bespoke code has been eliminate from both the Hub and Spoke classes and is now in SPCommunicator
5. Automatic checking for sanity on buffer lengths has been moved to the class FieldLengths, which centrally stores the length of each field. Validation is done automatically and generically in SPCommunicator.register_recv_field
6. The SPCommunicator classes now register as a class attribute what they send and receive. This isn't necessary, but makes it possible to write the method SPCommunicator.register_receive_fields in a generic way. But maybe we should just be doing calls to super() with traditional inheritance instead?
7. Improves termination time: each call to Spoke.got_kill_signal only checks the shutdown field. Further, instead of all cylinder ranks agreeing on shutting down by checking is_new, we now terminate when the first one gets the shutdown indicator through an SPCommunicator.allreduce_or call to synchronize the action. This results in (a) faster termination and (b) more flexibility: Spoke.got_kill_signal can be called multiple times to jump out of expensive function executions.

[Copilot]

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.