GeezerVison

…peline.

The hosts of a multihost pipeline exchange data over raw TCP.  To move that
to TLS, each host needs a client and a server configuration that
authenticate the other host as a host of the same pipeline.

The identity to check is already in each pipeline certificate.  An X.509
certificate lists the names that it is valid for in its "subject
alternative name" (SAN) extension.  Most SAN entries are DNS names or IP
addresses, which name a network location, but an entry can also be a URI.
The runner gives each pipeline certificate the URI entry
`urn:feldera:pipeline:<pipeline id>`.  Because the private CA signs it, a
peer that completes a TLS handshake with that certificate proves that it
belongs to that pipeline, wherever it runs.  The control plane's
certificate has `urn:feldera:control-plane` instead.  `Caller` reads these
URIs, and the compiler, runner, pipeline and coordinator servers already
use it to admit clients.

`PipelinePeerTls::from_chain` builds both configurations from the pipeline
certificate chain and its key.  Both ends present the pipeline
certificate.  Both trust only the private CA, the first certificate after
the leaf, as the existing client certificate verifier does.  Both admit
only a peer whose certificate has the same pipeline URI, so the control
plane, external clients and other pipelines are refused.  The server
requires a client certificate.

The client needs a custom server certificate verifier, which rustls allows
only through `ClientConfig::dangerous()`.  The built-in verifier checks
that a server's DNS name or IP address entries include the name that the
client connected to.  Hosts connect by pod IP address, which no
certificate lists, and the pipeline URI is the identity that matters.  The
custom verifier does what the built-in one does without CRLs (chain to the
private CA, validity period, server auth extended key usage, handshake
signatures), except that it checks the pipeline URI instead of the name.
Two alternatives avoid `dangerous()`, but each is worse:

- Verify against a DNS name that matches the wildcard DNS entry of our own
  certificate, `*.<pipeline service>.<namespace>.svc.cluster.local`.  This
  makes the runner's DNS naming the identity, separate from the pipeline
  URI that the servers check.

- Verify against a name that every pipeline certificate has, such as
  `localhost`, and check the peer's pipeline URI after the handshake.  This
  splits verification in two, and the connection is only safe if every
  caller remembers the second check.

The tests run in-memory handshakes.  The client and the server side are
each checked on their own, against another pipeline, the control plane, a
sibling CA, the root CA, and no certificate.  Each test fails if the check
it covers is removed.

Signed-off-by: Ben Pfaff <[email protected]>

`CircuitConfig::exchange_tls` holds rustls client and server
configurations.  When it is set, every exchange connection between hosts
starts with a TLS handshake, and the exchange magic and messages go inside
TLS.  Without it, there is no TLS.

The server now sends the exchange magic back after it reads the client's,
and the client waits for it before it uses the connection.  This is how
the client learns that the server accepted its certificate: in TLS 1.3,
the client finishes its handshake before the server checks the client
certificate, so a refused certificate shows up only when the client next
reads.  Without the reply, `connect` succeeded, the refusal arrived later
as a dropped connection, and the client reconnected at once, in a loop of
full handshakes.  With it, a refused certificate in either direction fails
`connect`, which waits for the backoff before it tries again and logs a
warning.  The reply is also sent without TLS, so the magic's version
becomes v2.

On both sides, the TLS handshake and the magic exchange each time out
after 10 seconds, so a peer that stalls, or a stray client that connects
and sends nothing, does not keep a task forever.  The server does the
handshake in the task for the connection, not in the accept loop, so a
slow client cannot hold up other connections.  It logs
refused and timed-out handshakes with the peer address, as it does for a
bad exchange magic.

tokio-rustls can return from a write before the encrypted bytes reach the
socket, so both sides flush after the magic, the client after each
message and each ping, and the server after each acknowledgement.  Without
TLS, a flush does nothing.  Plain TCP connections still use `into_split`.

The new dev tweak `tls_exchange`, on by default, turns exchange TLS off,
so that data goes in plaintext even when `exchange_tls` is set.  On
localhost, with 2 hosts of 4 workers, TLS did not change the rate of
rounds with small messages, and it lowered the throughput of 1 MiB
messages by about 29%, from about 1,150 MB/s to about 820 MB/s.

The tests run the multihost exchange tests over TLS with fault injection,
so reconnections go through the handshake.  A handshake counter shows that
TLS was used.  Other tests check that a TLS client does not send the magic
in plaintext, that a TLS server closes a plaintext connection, that
`connect` fails when the server refuses the client's certificate, that
`connect` succeeds once the server answers the magic, and that the server
closes a connection that sends no magic, and that exchange does not use
TLS with the dev tweak off.  Tests use a 2-second timeout.
Each of these tests fails if the behavior it checks is removed.

Signed-off-by: Ben Pfaff <[email protected]>

With `--require-client-certificates` (`FELDERA_REQUIRE_CLIENT_CERTIFICATES`,
Helm `mtls.enabled`), the pipeline loads its certificate and key from
`--https-tls-cert-path` and `--https-tls-key-path` and builds
`PipelinePeerTls`.  The controller passes it to the engine in the new
`CircuitParams::exchange_tls`, and the static engine sets it in dbsp's
`CircuitConfig`.  So the hosts of a multihost pipeline exchange data over
TLS and admit only hosts of the same pipeline.  Without client
certificates, exchange stays plaintext.

`CircuitParams` carries the `feldera-mtls` type, not the dbsp one, because
the engine interface does not depend on dbsp.  An engine that destructures
`CircuitParams` must name the new field.

The three `ControllerBuilder::open_*` methods each passed the builder's
settings on to `ControllerInit` with the same chain of calls.  They now
share `ControllerBuilder::finish`, so that a setting cannot reach some of
them and not others.  A new test checks that opening a controller passes
the exchange TLS on to the circuit, and fails if `finish` drops it.

Document it in the HTTPS guide, the Helm chart reference and the changelog.

Signed-off-by: Ben Pfaff <[email protected]>

The peer configurations negotiate only TLS 1.3, but the rustls verifier
traits still require `verify_tls12_signature`, and both verifiers
implemented it by checking the signature.  Now both refuse, so a TLS 1.2
handshake fails even if the protocol versions change.

`PipelinePeerTls::from_chain` errors now say what is wrong in the setup:
a private key that does not match the pipeline certificate, or a
certificate file without the CA chain, which the runner appends from the
private CA in the Helm value `caSecretRef`.

`peer_tls_refuses_tls12` checks that a host refuses TLS 1.2 as client
and as server.  It fails if the configurations allow TLS 1.2.
`peer_tls_needs_pipeline_chain` now also checks the error for a key that
does not match the certificate.

Signed-off-by: Ben Pfaff <[email protected]>

When the exchange TLS configuration cannot be built, the panic now names
the certificate and key paths, and says that the runner writes them from
the pipeline certificate, which it signs with the private CA in the Helm
value `caSecretRef`.

Signed-off-by: Ben Pfaff <[email protected]>