Automatically monitor and restart apache

14 Feb

Recently, we had an issue with apache that was causing it to refuse connections after running for awhile. It was randomly segfaulting and leaving child threads active, which meant that it’d eventually hit the MaxThreads directive and, you guessed it, refuse connections.

While we have not figured out exactly what’s causing it to segfault, we have come up with a temporary solution. We’ve written a PHP script that tries to connect to localhost with curl, and then a shell script that runs it and parses it’s result. The shell script is run with cron every five minutes.

Here’s how it all works:

Step 1 – Make the scripts

apache_check.sh – Put this somewhere like /root/scripts/apache_check.sh

servercheck.php – Put this somewhere like /var/www/servercheck.php

Step 2 – Configure and test
Make the shell script executable

And now test the script
Run the following commands, and you should see something like:
  Thu Feb 14 01:30:02 CET 2008 [APACHE OK]

Step 3 – Add to cron
Type the following command to enter the crontab editor

I will assume that it opened crontab in vi/vim for the next step.
Hit the ‘i’ key which will put you in insert mode, and type something similiar to:

Hit esc, then type :wq and press enter. You should see something like:
  crontab: installing new crontab
This will run the script every 5 minutes. For more information on crontab, follow this link: http://www.adminschoice.com/docs/crontab.htm

Step 4 – Verify the cron job
Type the following command to list the current cronjobs

If you do not see the line you entered in the previous step, something went wrong. Go and try it again!

Step 5 – Verify the script is working
Type the following command and wait for at least five minutes, and you should see a new line appear:

If you have followed all those steps successfully, congratulations! Apache will now restart if it runs out of available threads and it’ll email you to let you know what it’s done.

Also, if you know of a nicer way to display commands (a plugin like the syntax highlighter?) please let me know.

2 Responses to “Automatically monitor and restart apache”

  1. DM May 7, 2008 at 10:09 pm #

    You could also use ‘monit’ – it has many options, and it saves a lot of time while trying to monitor many daemons.

  2. Alex August 16, 2008 at 3:42 pm #

    Your blog is interesting!

    Keep up the good work!

Leave a Reply