On EC2 this post wouldn’t exist. You’d attach a role to the instance, the instance would pick up temporary credentials from the metadata service, they’d rotate themselves, and you’d never see a key.
Lightsail doesn’t do that. There’s no instance profile, no --iam-instance-profile on create-instances, nothing. Lightsail is AWS with the scary bits taken off, and it turns out the scary bits include the good bits.
That mattered for the fleet migration (moving a lot of WordPress sites off Bitnami Lightsail onto AWS’s managed blueprint; the pillar post has the full picture). The site archive travels old server → S3 → new server, so both servers need to run aws s3 cp. And aws s3 cp needs credentials from somewhere.
This is how I did it, and then the part where I reread my own code while writing this and found I’d been less careful than I thought.
The shape of it
- One IAM user per AWS account, used only by the migration tooling. No console login.
- One inline policy on that user, scoped to what the pipeline actually does.
- The access keys live in
~/.aws/credentialson the operator laptop, as a named profile, and nowhere else. - When a server needs to touch S3, the credentials ride along on that one SSH command as environment variables. Nothing is written to the server’s disk.
A user rather than a role, because there’s nothing to attach a role to. You could have the laptop assume a role and pass on those temporary credentials instead, and I’ll come back to that, because it’s where this ends up.
The policy
It started life as “Lightsail full access plus S3 full access”, which is how these things always start, and got cut down to the calls each phase makes. With placeholder names:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadFleetState",
"Effect": "Allow",
"Action": ["lightsail:GetInstance", "lightsail:GetInstances", "lightsail:GetStaticIps", "lightsail:GetBundles", "lightsail:GetBlueprints"],
"Resource": "*"
},
{
"Sid": "ProvisionAndCutover",
"Effect": "Allow",
"Action": ["lightsail:CreateInstances", "lightsail:PutInstancePublicPorts", "lightsail:DetachStaticIp", "lightsail:AttachStaticIp", "lightsail:StopInstance"],
"Resource": "*"
},
{
"Sid": "TransportBucket",
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": "arn:aws:s3:::example-migrations-*"
},
{
"Sid": "TransportObjects",
"Effect": "Allow",
"Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
"Resource": "arn:aws:s3:::example-migrations-*/*"
}
]
}
The things left out matter as much as what’s in. No s3:CreateBucket, no lifecycle configuration, no lightsail:Delete*, and nothing in iam:* at all. Creating the bucket and its lifecycle rule is a one-off admin job per account. The day-to-day identity doesn’t get to do it. The worst it can do to the old server is stop it, not delete it.
The bucket is named with the account ID on the end so it’s globally unique, and the policy matches on the prefix. It also has a lifecycle rule that expires objects after a few days, which is the backstop for when the cleanup step doesn’t run.
Passing credentials without leaving them behind
On the laptop, the transfer script reads the keys straight out of the profile:
$key = & aws configure get aws_access_key_id --profile $profile $secret = & aws configure get aws_secret_access_key --profile $profile
Then every remote command goes through a helper that prefixes the environment, base64-encodes the whole lot so nothing gets mangled by quoting between PowerShell, plink and bash, and pipes it into bash on the far side:
$envPrefix = ''
foreach ($k in $EnvVars.Keys) {
$v = $EnvVars[$k] -replace "'", "'\''"
$envPrefix += "$k='$v'; export $k; "
}
$payload = [Convert]::ToBase64String([Text.UTF8Encoding]::new($false).GetBytes($envPrefix + $bashCmd))
$remoteCmd = "echo $payload | base64 -d | bash"
Upload from the old box, download on the new box, delete the object from the laptop. The credentials exist on each server for the length of one SSH session and are never written to a file there.
My runbook describes these as “short-lived credentials”. While writing this I had to stop and admit they aren’t. The call is short-lived. The keys live until I rotate them.
What I got wrong
Three things, now I look at it properly.
Base64 isn’t hiding anything. It’s there for quoting, not secrecy, and it sits in the command line. While that transfer runs, the payload is in the argument list of a shell process on the server, and on a stock Linux box any local user can read other users’ command lines out of /proc. That includes the web server user, which is the user a compromised plugin would be running as. The window is a minute or two. It’s still a window.
The servers get far more than they need. A server running aws s3 cp needs GetObject or PutObject on one bucket. What it’s handed is the laptop’s whole identity: create instances, detach static IPs, stop instances, across the account. Leaked from one box mid-transfer, that key could take other sites offline.
Nothing expires. If a key does leak, it’s valid until someone notices.
What I’m changing it to
All three have reasonably small fixes, and they stack.
Send the payload over stdin instead of the command line. plink passes its own standard input through to the remote command, so the remote side becomes plain bash and the script goes in on stdin, which doesn’t show up in /proc/*/cmdline.
And stop sending the real keys at all. Have the laptop mint temporary credentials, scoped down to the one bucket, and send those. For an IAM user, sts get-federation-token does exactly this. You pass a policy, and the temporary credentials get the intersection of that policy and the user’s own:
aws sts get-federation-token \
--name site-transfer \
--duration-seconds 900 \
--policy '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":"arn:aws:s3:::example-migrations-123456789012/*"}]}' \
--profile migrator
The migration user needs sts:GetFederationToken added to its policy for this, which is the only permission the change adds. That hands back an access key, a secret and a session token. Pass all three as environment variables (the CLI needs AWS_SESSION_TOKEN too) and the box gets fifteen minutes of access to one bucket and nothing else. If it leaks, it’s useless to anyone by the time they’ve worked out what it is.
Fifteen minutes is the shortest duration STS will issue, which is plenty for a transfer. Some of the big multisite archives might push past it, so those get a longer duration, not a broader policy.
The housekeeping bit
AWS hands you a new user’s access key once, as a CSV download. Put it straight into ~/.aws/credentials or a password manager and delete the file. It shouldn’t be sitting in your project folder, or your Downloads folder, or anywhere a sync tool or a stray git add . might pick it up. It’s very easy to leave one lying around.