Time for some ghetto monitoring
If you came here between April 28 and about an hour ago, you got a “couldn’t connect to database” error. Oops! Seems my limited memory EC2 instance got a little overwhelmed by php processes and decided the db server, drizzled, should die to make more room for PHP. Ooops! Time to drop pm.max_children.
I don’t have any monitoring setup for the site, so I just now figured it out. Until I get proper monitoring, I’ve installed this fancy bit of duct-tape upstart magic:
start on stopping
env | mail -s "$JOB is stopping!" firstname.lastname@example.org
What does this do? Well is emails me whenever upstart gives up respawning something, or I manually stop a service.
Its not monitoring. I need monitoring. But this is a nice little hack to prevent a regression while I figure that out.