@@ -208,34 +208,44 @@ You can connect to an external PostgreSQL database with the following steps: |
208 | 208 |
|
209 | 209 | ### Rotate the database password |
210 | 210 |
|
211 | | -You can change the password of the bundled database without losing any data. |
212 | | -Updating the secret alone isn't enough: Postgres only reads the password from the |
213 | | -secret the first time it starts on an empty volume, and the Feldera components only |
214 | | -read it when they start. So you change the password in Postgres itself, update the |
215 | | -secret to match, and restart the components that connect to the database. |
| 211 | +To change the password of the bundled database, updating the secret alone isn't |
| 212 | +enough: Postgres only reads the password from the secret the first time it starts |
| 213 | +on an empty volume, and the Feldera components only read it when they start. So |
| 214 | +you change the password in Postgres itself, update the secret to match, and |
| 215 | +restart the components that connect to the database. |
216 | 216 |
|
217 | 217 | The steps below assume the release name `feldera`, the namespace `feldera` and the |
218 | 218 | secret `feldera-db-secret` from |
219 | 219 | [Configure custom database credentials](#configure-custom-database-credentials). |
220 | 220 | Adjust them if yours are different. |
221 | 221 |
|
| 222 | +If you use the default `feldera-db-insecure-secret`, don't change that secret. |
| 223 | +The Helm chart owns it, so the next `helm upgrade` sets the old password back. |
| 224 | +Move to your own secret instead: in step 1 read the user and password from |
| 225 | +`feldera-db-insecure-secret`, in step 3 create `feldera-db-secret`, and in step 4 |
| 226 | +run `helm upgrade` with your usual values and |
| 227 | +`--set felderaDatabaseSecretRef="feldera-db-secret"` instead of the restart. |
| 228 | +The upgrade restarts the components with the new secret. |
| 229 | + |
222 | 230 | 1. **Choose a new password:** use only letters and digits. Feldera puts the password |
223 | 231 | into a `postgresql://` connection URL as is, so characters like `@`, `:` or `/` |
224 | | - break the connection. |
| 232 | + break the connection. Also save the current user and password, in case you need |
| 233 | + to go back. |
225 | 234 | ```bash |
226 | 235 | NEW_PASSWORD=$(openssl rand -base64 48 | tr -dc 'A-Za-z0-9' | head -c 32) |
| 236 | + DB_USER=$(kubectl get secret -n feldera feldera-db-secret -o jsonpath='{.data.\.user}' | base64 -d) |
| 237 | + OLD_PASSWORD=$(kubectl get secret -n feldera feldera-db-secret -o jsonpath='{.data.\.password}' | base64 -d) |
227 | 238 | ``` |
228 | 239 |
|
229 | 240 | 2. **Change the password in Postgres:** inside the database pod, `psql` connects over |
230 | 241 | the local socket, so you don't need the old password. The statement goes in on |
231 | 242 | stdin, which keeps the password out of the process list. |
232 | 243 | ```bash |
233 | | - DB_USER=$(kubectl get secret -n feldera feldera-db-secret -o jsonpath='{.data.\.user}' | base64 -d) |
234 | 244 | printf "ALTER ROLE \"%s\" PASSWORD '%s';\n" "$DB_USER" "$NEW_PASSWORD" | \ |
235 | 245 | kubectl exec -i -n feldera feldera-db-0 -- sh -c 'psql -U "$POSTGRES_USER" -d postgres -v ON_ERROR_STOP=1' |
236 | 246 | ``` |
237 | 247 | It should print `ALTER ROLE`. From now on, new connections with the old password |
238 | | - fail. Connections that are already open stay up, so continue with the next steps |
| 248 | + fail. Connections that are already open continue to work, so do the next steps |
239 | 249 | right away. |
240 | 250 |
|
241 | 251 | 3. **Update the secret:** |
@@ -244,9 +254,17 @@ Adjust them if yours are different. |
244 | 254 | --from-literal=.user="$DB_USER" --from-literal=.password="$NEW_PASSWORD" \ |
245 | 255 | --dry-run=client -o yaml | kubectl apply -f - |
246 | 256 | ``` |
247 | | - If the secret is managed by a secret manager, for example through the External |
248 | | - Secrets Operator, update the value there instead and wait until the Kubernetes |
249 | | - secret has the new value. |
| 257 | + Also update the place the secret comes from, for example your |
| 258 | + `feldera-db-secret.yaml` file or your GitOps repository. If someone applies the |
| 259 | + old value again later, the components can't log in after their next restart. |
| 260 | + |
| 261 | + If the secret is managed by the External Secrets Operator, update the value in |
| 262 | + your secret manager instead. The operator only syncs at its `refreshInterval`, |
| 263 | + so force a sync and make sure the status is `SecretSynced` before step 4: |
| 264 | + ```bash |
| 265 | + kubectl annotate externalsecret -n feldera feldera-db-secret force-sync=$(date +%s) --overwrite |
| 266 | + kubectl get externalsecret -n feldera feldera-db-secret |
| 267 | + ``` |
250 | 268 |
|
251 | 269 | 4. **Restart the Feldera components:** |
252 | 270 | ```bash |
@@ -263,8 +281,14 @@ Adjust them if yours are different. |
263 | 281 | authentication errors. Your pipelines keep running during the rotation. |
264 | 282 |
|
265 | 283 | If step 2 fails, nothing has changed yet, and the components still use the old |
266 | | -password. If you use an external database, change the password there, update |
267 | | -`.connection_url` in the secret, and do step 4. |
| 284 | +password. |
| 285 | + |
| 286 | +If step 3 or 4 fails, for example a pod stays in `CrashLoopBackOff` with |
| 287 | +authentication errors, go back to the old password: run steps 2 and 3 again with |
| 288 | +`"$OLD_PASSWORD"` in place of `"$NEW_PASSWORD"`, then do step 4. |
| 289 | + |
| 290 | +If you use an external database, change the password there, update |
| 291 | +`.connection_url` in the secret, and proceed to step 4. |
268 | 292 |
|
269 | 293 | ### Feldera installation overview |
270 | 294 |
|
|