In this article...

If they are not caught in review, keys might sit there for years with permissions that still work.

The alternative is short-lived credentials: tokens issued by the cloud provider that expire on their own and renew automatically. Below, we’ll look at where keys tend to hide on Linux hosts, what temporary credentials protect against, and walk through a real-world example of backup migration from a static AWS key to an EC2 instance role.

Where Are the Keys Actually Kept?

The AWS CLI stores keys in ~/.aws/credentials, a plain-text INI file. AWS SDK can read this file, depending on your configuration and the provider chain. Google Cloud’s tooling keeps its state under ~/.config/gcloud, and the Azure CLI caches tokens under ~/.azure. A downloaded Google service account key, on the other hand, is just a JSON file, and it lives wherever the person who downloaded it happened to save it – often in ~/Downloads. 

Then, there are environment variables. These became a popular way to pass configuration into applications, and secrets followed closely behind. An environment can be read through /proc//environ by the process owner and by root.

The systemd project’s own documentation on service credentials flags environment variables as a problem for secrets because they’re passed down the process tree by default. It doesn’t help that unit files in /etc/systemd/system are usually world-readable, so a key placed in an Environment= line can be visible to every local user on the machine.

The next location is Shell history. You can simply type export AWS_SECRET_ACCESS_KEY=... at a prompt. Unless HISTCONTROL is configured to ignore commands that start with a space, that secret is now sitting in ~/.bash_history.

Finally, cloud credentials regularly turn up in configuration management. That includes places like Ansible variable files that were supposed to be vaulted, Terraform .tfvars files, Docker Compose .env files, and Kubernetes secrets that are base64-encoded rather than encrypted (for Kubernetes, encryption at rest can be configured separately).

The Problem with Static Credentials

Static key copies will keep working until someone rotates, deactivates, or deletes the key.

These copies can multiply. Whenever the home directory gets backed up, the key goes in the backup. A dotfile gets committed to the wrong Git repository. A support bundle is sent to a vendor with the application’s config tucked inside it. Access controls for each copy are usually weaker than the server’s.

Keys are also easy to spot. AWS gives long-term access key IDs the prefix AKIA and temporary ones the prefix ASIA, and secret scanners, whether they’re run by GitHub or by attackers, look for exactly that pattern. Test keys deliberately leaked to public repositories have repeatedly been picked up within minutes. Attackers actively look for exposed keys. 

When nobody is sure which jobs still depend on a particular key, they hesitate to delete keys. On older systems, that creates a collection of credentials that nobody can account for. The risks of long-lived cloud credentials extend beyond configuration files: application memory dumps can expose keys that remain usable after the original leak.

Temporary Credentials: Better, But Not Bulletproof

In AWS STS, temporary credentials are made up of three parts – an access key ID, a secret, and a session token. Once their expiry time passes, all three are useless. However, while temporary credentials are valid, anyone holding all three parts can use them from another machine, with every permission the role grants. 

The best-known example is the 2019 Capital One breach, where a server-side request forgery (SSRF) through a misconfigured web application firewall was used to ask the EC2 metadata service for role credentials. Those credentials were perfectly legitimate and perfectly temporary, and they were still used to copy data out of S3. Shorter-lived doesn’t mean harmless; it just gives the attacker less time.

AWS responded later that year with version 2 of the Instance Metadata Service (IMDSv2). That reduces certain SSRF risks, but it isn’t complete protection: an attacker who can run code on the instance, or exploit a flaw that lets them send arbitrary requests, can still fetch a token just like any legitimate process. 

AWS also added condition keys specific to EC2 instance credentials, which allow a policy to check where a request using those credentials came from. However, those conditions only work if you explicitly write them into your IAM or resource policies, and if your workloads reach AWS through the network paths the policy expects (such as VPC endpoints). Without changing the default configuration, credentials lifted off the server can still be used from somewhere else.

If something does leak, administrators can revoke a role’s active sessions. That attaches a deny policy to every credential the role issued before a chosen point in time. The deny policy doesn’t distinguish between the attacker’s copy and the legitimate workload’s copy, so every process using that role loses access until it obtains credentials issued after the revocation.

A Practical Example: Moving a Backup Job to an Instance Role

An Ubuntu server on EC2 runs PostgreSQL, and a systemd timer starts pg-backup.service at 02:15 every night. The service runs as a local backup user, pipes pg_dump through zstd, and uploads the result to s3://example-backups/db/ with aws s3 cp. It authenticates with an access key belonging to an IAM user called backup-bot, stored in /home/backup/.aws/credentials and created a few years ago by an engineer who has since left the company. 

Before changing anything, the first step is to find out what the key actually does. On the host, grep -rIl "AKIA" /etc /home /opt /srv will list any files containing long-term key IDs, and it’s worth running across the whole fleet, because a key you find once is rarely only in one place. For each key, aws iam get-access-key-last-used shows when it was last used, in which region, and against which service. The IAM credential report, generated with aws iam generate-credential-report, gives you the same picture across every user in the account, key ages included.

CloudTrail can fill in the gaps, with one caveat: aws cloudtrail lookup-events can filter by access key ID, but it only covers the last 90 days of management events. S3 object operations like PutObject are data events, and those are only recorded if a trail has been set up to capture them. So if the backup key’s activity looks suspiciously quiet, that’s probably why. 

Keep the Permissions Tight

Next, you’ll need an IAM role. It needs a trust policy that allows ec2.amazonaws.com to assume it, and a permissions policy that allows exactly what the job does and nothing more. In this case, that means s3:PutObject on arn:aws:s3:::example-backups/db/*.

The AWS CLI switches to multipart once a file goes over its default threshold of 8 MB, which a database dump almost certainly will, and the individual multipart operations are covered by s3:PutObject. Adding s3:AbortMultipartUpload lets the CLI tidy up after a failed upload, and a bucket lifecycle rule that aborts incomplete multipart uploads after a few days will catch anything left over. This matters more than it sounds: orphaned parts don’t show up in a normal bucket listing, but you’re still billed for them. If the bucket uses SSE-KMS encryption, the role will also need kms:GenerateDataKey on the key, plus kms:Decrypt for multipart uploads.

There’s no s3:DeleteObject, no s3:ListBucket, and nothing outside the db/ prefix. A backup server that can delete its own backups is a gift to ransomware, and with bucket versioning enabled, even an overwrite keeps the previous version intact. For retention that administrators can’t bypass, S3 Object Lock in compliance mode takes this a step further: once a retention period is set, no user, including the account’s root user, can delete or overwrite the protected object versions until it expires. Governance mode, by contrast, can be overridden by users with the right permissions, so it’s the compliance mode you want if the goal is backups nobody can remove early.

Attach the role with aws ec2 associate-iam-instance-profile, naming the instance and its instance profile, which is the container EC2 uses to carry a role (the EC2 documentation on IAM roles explains how the two fit together). While you’re there, it’s a good idea to require IMDSv2 with aws ec2 modify-instance-metadata-options --http-tokens required – but check first. EC2 publishes a CloudWatch metric called MetadataNoToken, which counts requests still using the older IMDSv1. If it’s above zero, something on the instance (often an old agent or an outdated SDK) will break the moment you switch v1 off.

Getting (and Renewing) the Credentials

The metadata service lives at 169.254.169.254, an address in the link-local range reserved for traffic that never leaves the local network segment. With IMDSv2, a client first requests a token by sending a PUT to /latest/api/token, then reads /latest/meta-data/iam/security-credentials/ followed by the role name. What comes back is a small JSON document containing an access key ID starting with ASIA, a secret, a session token, and an Expiration timestamp. 

Running the backup under a dedicated backup user is good practice, but it doesn’t, by itself, isolate the role’s credentials. The metadata service doesn’t check which Linux user is asking, so any process on the instance that can reach 169.254.169.254, whether that’s a compromised web application, a stray script, or another user’s cron job, can request exactly the same credentials. AWS’s documentation describes ways to limit this with local firewall rules (for example, an iptables rule that only lets specific user IDs reach the metadata service), but the more fundamental point is that an instance role is shared by everything running on that instance, so it’s best not to mix untrusted workloads with a privileged role on the same machine.

According to AWS’s documentation on retrieving instance credentials, new credentials are made available at least five minutes before the old ones expire. AWS CLI and SDKs handle retrieval and refresh.

The AWS CLI and SDKs work through a chain of credential providers, and which one wins depends on the active configuration: environment variables, a named profile, and settings in the shared config and credentials files are typically checked before the instance metadata service. So if the service’s environment still points the CLI at the old static key, whether through a profile, a variable, or the files in /home/backup/.aws, the role can be attached and never actually used. The fix is to move the old credentials file somewhere only root can read it (moving it rather than deleting it gives you a rollback option), remove any AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY lines from the unit file or its EnvironmentFile, and run systemctl daemon-reload.

Then, check who the job thinks it is, and do it from inside the backup service’s actual execution environment rather than your own shell. A sudo -u backup aws sts get-caller-identity is a useful first look, but it won’t pick up the Environment= or EnvironmentFile= settings from the unit, so it can report the role while the real job is still using the old key. A more reliable approach is to add a temporary drop-in with systemctl edit pg-backup.service containing ExecStartPre=/usr/local/bin/aws sts get-caller-identity (adjusting the path to wherever aws is installed), start the service and read the output in journalctl. You should see an ARN along the lines of arn:aws:sts::123456789012:assumed-role/pg-backup-writer/i-0abc..., with the instance ID on the end. If it still names the IAM user, the old key is still being picked up somewhere in the provider chain. Once you’re happy, remove the drop-in.

Testing Credential Renewal and Handling Failures

Start the service manually with systemctl start pg-backup.service, follow its output with journalctl -u pg-backup.service, and check that the backup has landed in the S3 bucket. That confirms the job can upload using its current credentials. It doesn't tell you whether those credentials will renew successfully.

A test only proves that one set of credentials works. The real test comes after the first credential expires and the AWS CLI has to retrieve another. Let the job run on its normal schedule for at least a week, checking that each backup completes and the expected objects appear in S3.

If renewal breaks, the job may only continue working until credentials expire. On Linux, the usual causes are:

• Firewall rules that block traffic to the instance metadata address at 169.254.169.254.
• Proxy settings that send metadata requests through a corporate proxy because NO_PROXY doesn't exclude 169.254.169.254.
• Containers that can't receive the IMDSv2 token response because the instance metadata hop limit is set to 1.
• Instance profiles that were changed, replaced, or detached during unrelated maintenance.
• Custom code that reads credentials once at startup and holds on to them, eventually failing with ExpiredToken errors instead of requesting fresh credentials.

If you have a staging instance, test renewal by adding a temporary nftables rule that blocks traffic to 169.254.169.254, wait for the current credentials to expire, and run the backup again. The job should fail with an error that your monitoring can detect. Remove the rule afterwards and confirm that the AWS CLI can retrieve credentials again.

Also check the system clock when authentication starts failing. AWS API requests are signed with timestamps, and a clock that drifts too far from AWS's time can produce errors such as RequestTimeTooSkewed. A badly configured clock can also interfere with how a client handles credential expiry and refresh, making a time synchronization problem look like a credential problem.

Run timedatectl to check whether the system clock is synchronized. On EC2, chrony can use the Amazon Time Sync Service at 169.254.169.123. If chrony has stopped syncing, fix that before spending hours investigating IAM policies or replacing credentials that are still valid.

Backup failure might not draw attention until someone needs to restore a database. An OnFailure= directive in the systemd unit can trigger a notification when the service exits with an error. The backup script should also use set -euo pipefail, since without pipefail, a failed pg_dump can be hidden by a successful downstream command, leaving a small or incomplete backup file while the service reports success.

Logs provide another useful signal. Search for Unable to locate credentials, ExpiredToken, and AccessDenied, but don't treat every authentication error as a renewal problem. AccessDenied, for example, can mean the role was retrieved successfully but lacks permission to write to the bucket.

The most reliable check watches the backup itself rather than just the service. A dead man's switch can send a heartbeat to a monitoring service after each successful run, or a scheduled check can verify that the newest object under db/ is less than 26 hours old.

That catches failures regardless of their cause. Expired credentials, blocked metadata requests, a drifting clock, a full disk, or an unavailable database can all leave the bucket without a fresh backup. If the expected backup doesn't appear, someone needs to know before the next restore.

Retiring the Old Key

Once the role has proven itself, it’s time to retire the old key. Do this in two stages, because only one of them can be undone. First, deactivate it with aws iam update-access-key --user-name backup-bot --access-key-id AKIA... --status Inactive. Deactivation can be reversed in seconds, which makes it the safest move when you’re not 100% sure what the key was doing. 

After that, leave the key inactive through at least one full cycle of every schedule that might plausibly use it, keeping an eye on CloudTrail and your application logs for authentication failures. If nothing unexpected happens, then it’s okay to delete the inactive key with aws iam delete-access-key. If backup-bot has no other keys and no other purpose, delete the user as well, and destroy the old credentials file you moved aside earlier. Running a fresh credential report afterward is a good way to close the loop.

Recent Searches