Two small sites went down on a Friday and the Apache process was still running on both. It just wasn’t answering anyone.
The error log had one line in it, repeated about once a second:
AH03490: scoreboard is full, not at MaxRequestWorkers
Which is a lovely message if you enjoy contradictions. Full, but not at the limit. I looked at it for a while.
Three days earlier
The previous Monday I’d rolled the whole fleet from mod_php over to PHP-FPM with the event MPM. Smaller memory footprint, and Apache stops dragging a PHP interpreter into every child process. I’d sized the MPM settings by box tier, with the nano boxes at the bottom. The config for each tier looked roughly like this:
<IfModule mpm_event_module>
ThreadLimit 25
ThreadsPerChild 25
ServerLimit 1
MaxRequestWorkers 25
MaxConnectionsPerChild 500
</IfModule>
ServerLimit was worked out as MaxRequestWorkers / ThreadsPerChild. Twenty-five workers, twenty-five threads per child, so one child process. The sum is right. It’s also how the box ended up with no room to move.
What the scoreboard actually is
Apache keeps a table of its child processes in shared memory, called the scoreboard, and ServerLimit sets how many rows it has. With the event MPM, a child that’s been told to exit doesn’t vanish at once. It stops accepting new connections, finishes the ones it has, and keeps its row until they’ve drained. Keep-alive connections can take a while to drain.
Now there’s a reason for a child to be told to exit. Two of them, actually. A graceful reload does it. So does MaxConnectionsPerChild: after 500 connections the child retires and gets replaced.
If your only row is occupied by a child that’s on its way out, there’s no row for the replacement. Apache can’t start a new child, so it can’t serve anything new, and it says so every second until someone intervenes. “Scoreboard is full” means every row is taken. “Not at MaxRequestWorkers” means the worker count hasn’t hit its cap. Both are true. They’re just measuring different things.
I had a limit of one and a recycle rule that retired that one child every 500 connections. It’s a design that works right up until a busy moment.
How bad was it
I went and checked the rest of the fleet. Fifty-two of fifty-eight boxes had logged that line since the rollout. Only two had fallen over completely, which says more about luck than about the config. The other fifty had been one slow keep-alive away from the same thing.
Both of the dead boxes needed a full restart, not a reload. More on that below.
The fix
Give the scoreboard spare rows. A row that isn’t in use costs close to nothing, and MaxRequestWorkers still caps actual concurrency, so a bigger ServerLimit doesn’t mean more load.
MPM_SRV_LIMIT=$(( MPM_SL * 3 )) [ "$MPM_SRV_LIMIT" -lt 4 ] && MPM_SRV_LIMIT=4
Three times the number of children the worker count needs, with a floor of four. The nano boxes go from one row to four.
The recycling rule changes too:
MaxConnectionsPerChild 10000
That setting made sense under mod_php, where children carried a PHP interpreter and leaked memory slowly, so retiring them regularly was useful housekeeping. Under FPM the Apache children don’t run PHP at all. Retiring them every 500 connections buys nothing and, as it turns out, keeps creating the exact gap that wedged the boxes.
The part that bit me twice
You’d think changing the config and reloading would do it. It doesn’t. ServerLimit is only read on a full restart. A graceful reload keeps the old value, so my first attempt at “fixing” a wedged box with a reload left it exactly as wedged as before.
And the reload was itself the thing that tipped the box over, because it asks the existing child to exit with nowhere to put a replacement.
The retrofit script now compares the MPM config it wants against what’s on the box. If they differ, it does a proper restart. If they match, a reload is fine.
if [ "$MPM_CHANGED" = 1 ]; then
systemctl restart apache2
else
systemctl reload apache2
fi
It went out to 56 boxes that same day.
What I’d take from it
I had the arithmetic right and the model wrong. MaxRequestWorkers / ThreadsPerChild is the minimum number of rows you need if nothing ever exits. Something always exits.
If you’re running the event MPM on a small box, check ServerLimit against your worker settings. If it’s exactly the quotient and nothing more, you’re one reload from this.