Managing Long-Running Processes with Supervisor

28 July 2012

Supervisor is a great tool to start and manage long-running processes and the log files associated with them. It offers a lot of helpful features and is easy to get up and running.

Install & Start Supervisor

Supervisor works out of the box, so this part is easy.

sudo apt-get install supervisor -y && sudo supervisord

Our First Program

Supervisor looks for configuration files in /etc/supervisor/conf.d/ and it is a good idea to keep a single conf file per process. To start, let's set up a simple long-running python script. Don't pay too much attention to the contents of the conf file, we'll go over that later.

Create /etc/supervisor/conf.d/test_python.conf containing:

[program:test_python]
command=python -u test.py
directory=/home/ubuntu
stdout_logfile=/home/ubuntu/test_python_output.txt
redirect_stderr=true

Create ~/test.py containing:

import time
while True:
	print(time.ctime())
	time.sleep(1)

When creating the conf file, be sure to replace /home/ubuntu both times with the path to your home directory (~ isn't allowed). If you run test.py manually you'll see it prints out the time once per second.

Using SupervisorCTL

SupervisorCTL is the tool you will use to manage everything in Supervisor. Start by running sudo supervisorctl and then typing reread. The reread command goes through the conf.d directory and loads any new program conf files. Once you see the new test_python program available, you can use the add command to load and start it (add test_python). The status command shows you all running processes, their PID and how long they've been running. Here is a short list of my commonly used supervisorctl commands and what they do.

• reread - Reload all program conf files from the conf.d directory.
• add [program_name] - Add a newly created conf file to Supervisor and start the process.
• status - Check the status of all programs currently managed by Supervisor.
• start [program_name] - Start the given program. Used often with one-time scripts.
• restart [program_name] - Restart the given program.
• tail -f [program_name] - Watch the log file in real-time (same as UNIX tail -f [filename]).
• help - List all available commands.

Exit supervisorctl and check your log file (~/test_python_output.txt) to see that it is still running. If it isn't, you may have missed a step.

Program Configuration Files

There is a wide array of options you can use while setting up your program configuration file. I've listed my favorite options below. For a full list of commands and their parameters, check out the settings documentation.

• command is the command to run.
• directory is the directory to run the command in.
• numprocs will automatically start N instances of that process.
• autostart is a boolean which tells whether to run automatically or not when Supervisor starts.
• autorestart will automatically restart a program that dies, under specific conditions (see docs).
• stdout_logfile is where to log the output of your program.
• redirect_stderr tells supervisor whether to include stderr in your standard output file.
• stdout_logfile_maxbytes is the size a log file is allowed to grow to until it is auto rotated.
• stdout_logfile_backups is the number of rotated log files to keep.

Web Interface

Supervisor provides a really nifty web interface to monitor and restart your processes. Just update the following file, restart supervisor, and you'll see it on port 9001. Feel free to change the port and protect it using HTTP Auth or whatever you see fit.

Add the following to /etc/supervisor/supervisord.conf

[inet_http_server] 
port=*:9001

One-Time Scripts

I've found that Supervisor also works great for running commonly used scripts that have multiple parameters or large log files. Be sure to set autostart and autorestart in the conf file to false, so your script doesn't end up running repeatedly.

Examples

MongoDB

[program:mongod_member]
command=/home/ubuntu/mongodb-linux-x86_64-2.0.6/bin/mongod --dbpath /mnt/mongo/ --replSet mongo5 --nojournal
directory=/home/ubuntu
stdout_logfile=/mnt/log/mongod.log
redirect_stderr=true