GeezerVison

An exchange client task connected to its peer forever, even after its
runtime and its peer stopped.  In the tests, it tried again about once
per millisecond, and it could connect to a new runtime that listened on
the old address of its peer (see issue #7800).

Now, after its `ExchangeClient` drops, the task sends only the messages
that the peer has not acknowledged, because the peer can still wait for
them, and then stops.  It also stops at the first failure to connect or
send.  It does not stop at once, because the connection uses zero
linger, so closing it would discard data that is not sent yet.

This problem is likely to affect only the tests, which run many
multihost runtimes in one process.  In production, each host runs in a
process of its own, which a restart replaces, so the old clients stop
when their process exits.

Signed-off-by: Ben Pfaff <[email protected]>

An exchange connection did not identify the runtime that it belongs to.
Thus, a client of another runtime could connect to the address of this
runtime and deliver its messages here.  This made
`sharded_accumulator::tests::take_received` fail with
`UnknownExchange(0)` (issue #7800): a client of an earlier test
connected to a port that the kernel gave to a host of a new test.  If
the exchange ID had existed, the message would have been delivered as
data without an error.

This problem is likely to affect only the tests.  It needs a client to
outlive its runtime in the same process, and a new runtime to listen on
the old address.  The tests run many multihost runtimes in one process,
on ports that the kernel selects and can reuse.  In production, each
host runs in a process of its own, which a restart replaces, so the old
clients stop when their process exits.  The previous commit makes the
first condition less likely, because a client now stops after its
runtime is gone, but a client can still connect while it sends its
last messages.

To fix the problem, `CircuitConfig` now has `exchange_uuid`, which every
host of a run shares.  A client sends it after the exchange magic.  The
server sends the magic back only if the UUID is its own, so a client of
another runtime fails to connect and waits for the backoff before it
tries again, and it cannot deliver or take over the slot of a real
peer.  A pipeline uses its deployment ID.  The test hosts use a new
random UUID.  The magic is now version 3.

Fixes #7800.

Signed-off-by: Ben Pfaff <[email protected]>

After its runtime was gone, an exchange client sent the messages that
its peer had not acknowledged, but it gave up at the first failure to
connect or send, and dropped the messages without a log line.  A peer
can still need them: `RuntimeHandle::join` drops the clients of a host
when the workers of that host finish, even while the workers of the
other hosts run.  In the multihost tests, which inject a failure into
about 1% of the sends, this probably made a peer hang in about 1 run
in 13.

Now, after the stop, the client still connects again when a connection
fails, until the peer acknowledges all of its messages.  It gives up
after 60 seconds, or 10 seconds in the tests, and then logs a warning
with the number of messages.  The check of the exchange UUID makes the
reconnection safe: if another runtime listens on the old address, the
client cannot deliver there.

This commit also makes the tests accept a ping before the client closes
the connection.

Signed-off-by: Ben Pfaff <[email protected]>