“SSH works, so the server’s ready.” I’ve believed that for years and I’m not proud of it.
The first wave of the Lightsail migration was seven boxes. Four of them hung. Not an error, not a failed step, just a launch script sitting there until the 1800-second bootstrap timeout ran out and killed it.
What a hung box looks like
The script’s job is to take a freshly created instance and tune it: timezone, MariaDB settings, Apache workers, PHP limits, the lot. Several steps restart MariaDB along the way. On four of the seven, one of those restarts never came back.
When I logged in, the database unit was in one of two bad states. Either it was stuck in deactivating (final-sigterm), or there was an orphaned mariadbd process still sitting in the unit’s control group, so every attempt to start the service failed with “Device or resource busy”. Systemd wouldn’t stop it, wouldn’t start a new one, and wouldn’t say anything useful about it.
The other two things waiting on the database, my script and the blueprint’s own installer, both waited politely until the clock ran out.
Two scripts, one database
Here’s what I’d missed. The new instance comes from an AWS-managed blueprint, and the blueprint does its own setup on first boot. That runs through cloud-init, in the final stage, and it installs WordPress and brings the database up as part of it.
The instance accepts SSH well before that finishes. So my script logged in, saw a perfectly responsive box, and started tuning. It restarted MariaDB while the blueprint’s installer was still in the middle of starting it. Twice, in fact, because there were two restart points in the script.
Two things racing to stop and start the same service is how you get a unit that’s half dead. It’s not a bug in MariaDB or in systemd. Two scripts that didn’t know about each other.
Why only four of seven? Timing, I assume. The three that survived presumably got their restarts in either before or after the installer’s critical moment. I can’t prove that and I’d rather say so than invent a tidy story.
Wait for the right thing
cloud-init has a command for exactly this, and I’d never needed it before.
wait_for_cloud_init() {
command -v cloud-init >/dev/null 2>&1 || return 0
echo "Waiting for first-boot setup to finish"
if sudo timeout 900 cloud-init status --wait >/dev/null 2>&1; then
echo " cloud-init: $(cloud-init status 2>/dev/null | awk '{print $2}')"
else
echo " still not done after 900s, continuing" >&2
fi
}
--wait blocks until cloud-init says it’s done. The timeout 900 is there because a cloud-init that never settles shouldn’t stop the whole launch; I’d rather proceed with a warning than hang for half an hour. The command -v guard is for other box types that don’t have it.
This runs before the script touches anything.
And a restart that can’t hang
Waiting removes the race I know about. It doesn’t remove a wedged unit for any other reason, so the restarts themselves needed a way out.
restart_mariadb() {
if sudo timeout 120 systemctl restart mariadb 2>&1 | tail -3 \
&& systemctl is-active --quiet mariadb; then
return 0
fi
echo " restart did not complete, forcing it" >&2
sudo systemctl kill -s SIGKILL mariadb 2>/dev/null
sleep 3
sudo systemctl reset-failed mariadb 2>/dev/null
sudo timeout 120 systemctl start mariadb 2>&1 | tail -3
systemctl is-active --quiet mariadb
}
Bound the restart at two minutes. If the service isn’t active afterwards, SIGKILL everything in the unit, which clears out that orphaned process, reset the failed state, and start it fresh.
One small thing about that first if. There’s a pipe in it, and without pipefail the exit status of the left side is tail‘s, which is always zero. So the part doing the real work is systemctl is-active. It’s fine here, but I wouldn’t have wanted to rely on the pipe’s status and I left a note saying why.
Both restart points in the script now go through the helper.
The takeaway
A server that answers SSH is a server that’s up. It isn’t necessarily a server that’s finished. If something else on the box is still setting itself up, and cloud-built images very often are, anything you do to shared services is a bet on timing.
Ask the machine whether it’s done. Then, because it might not answer, put a time limit on the asking.
