You can install a lot of djb software without having to learn the details of daemontools. Used in this way, daemontools is like a black box, handling control and logging of server processes reliably, quietly, although a little mysteriously.

To gain the most from daemontools, though, a little knowledge will go a long way. Then, before you know it, you will be looking for more services to run with it.

As a first example, we will consider blabbyd, a background process that periodically emits some useless message into a file. The executable script for such a daemon might look like this:

#!/bin/sh
# blabbyd.sh: a blabby daemon
MESSAGE="Hello, world!"
PAUSE=4
echo `date`: "starting blabbyd..." >> ./LOG
# loop forever
while :
do
    echo `date`: "blabbyd: ${MESSAGE}" >> ./LOG
    echo `date`: "blabbyd: sleeping for ${PAUSE} seconds..." >> ./LOG
    sleep ${PAUSE}
done
### that's all, folks!

Create the script in a text editor and save it in a file named blabbyd.sh. Then make the script executable:

# chmod 755 blabbyd.sh

Now run the script as a background process from the command line:

# ./blabbyd.sh &

Every 4 seconds this script will spit out a date(1) stamp and message to a file named LOG in the current directory. This file will grow forever with messages as long as blabbyd continues to iterate. Open another terminal and follow the log with tail(1):

# tail -f ./LOG

To make it stop, grep the process ID (PID) and kill it:

# ps ax | grep blabbyd
# kill XXX

Now consider: if you now wanted to run this same script as a background daemon whenever your server was running, you would have to make a number of arrangements:

• Figure out how to start this script with each system boot, perhaps editing the rc* scripts on some BSD systems, or the /usr/local/etc/rc.d/* scripts on others, or setting up System V init scripts in various places on Linux systems.

• Start and stop the daemon for maintenance, perhaps grep-ing the output from ps (as we did above) to determine the daemon's process ID (pid) for sending kill signals.

• Decide how to handle logging for the daemon, regulating the size of the log file, and recycling older log archives.

All of these issues, and many more, are easily and consistently handled by daemontools. As an example, we will set up the blabbyd daemon as a daemontools service in 3 simple steps:

1. First, create a directory for the service definition:

   # mkdir -p /var/svc.d/blabbyd
   

   For this example, copy the blabbyd.sh executable from above into the new directory:

   # cp blabbyd.sh  /var/svc.d/blabbyd
   

2. Next, create a run script for the service:

   ---

   #!/bin/sh
   # blabbyd/run
   # run script for "blabby" daemon:
   exec ./blabbyd.sh
   # that's all, folks! 
   

   ---

   Save the file to /var/svc.d/blabbyd/run and make it executable:

   # chmod 755 /var/svc.d/blabbyd/run
   

3. Finally, activate the service by linking into the /service directory:

   # ln -s  /var/svc.d/blabbyd  /service/blabbyd
   

That's it! svscan will now "see" the new entry within /service, and within 5 seconds will start the blabbyd service for you automatically.

Verify the service is running with svstat:

# svstat /service/blabbyd
/service/blabbyd: up (pid 451) 13 seconds

From a separate terminal window, follow the LOG file:

# tail -f /service/blabbyd/LOG

The display will show the blabbyd messages as they appear every few seconds. Then return to the control terminal window and use the daemontools svc -d utility to shut the service down:

# svc -d /service/blabbyd
# svstat /service/blabbyd
/service/blabbyd: down 7 seconds, normally up

Switch back to view the LOG file and notice that messages have stopped.

Bring the service back up again with svc -u:

# svc -u /service/blabbyd
# svstat /service/blabbyd
/service/blabbyd: up (pid 513) 3 seconds

Pause the service with svc -p:

# svc -p /service/blabbyd
# svstat /service/blabbyd
/service/blabbyd: up (pid 513) 29 seconds, paused

Continue the service with svc -c:

# svc -c /service/blabbyd
# svstat /service/blabbyd
/service/blabbyd: up (pid 513) 37 seconds

Send the service a TERM signal with svc -t; the service will terminate and then restart:

# svc -t /service/blabbyd
# svstat /service/blabbyd
/service/blabbyd: up (pid 747) 2 seconds

And so forth. With very little effort you have created a background daemon that:

• is easily and portably installed
• starts automatically with every system boot
• is automatically restarted if it should die
• has facilities that simplify stopping, starting, and restarting
• is easily signaled and monitored
• is easily removed

As for the last point, you will eventually have had enough of blabbyd and want to shut her up. Remove the daemon by deleting the symlink from /service, and then bring the service down completely:

# cd /service/blabbyd
# rm /service/blabbyd
# svc -dx .

This is the standard djb idiom for removing a service.

#!/usr/local/bin/bash

So far the blabby daemon has illustrated the minimal characteristics of a service running under daemontools:

• a service--some executable program or script--that runs in the foreground

• an executable script named run that exec's the service

• a symlink to the directory containing the run script from the /service directory monitored by svscan

In addition to the basic operations shown with blabbyd, daemontools also offers a number of other features that can be added to services, including:

• automatic logging service
• parameterization through environment variables
• assignment of user/group id priveleges for security

We'll rewrite the blabbyd daemon to take advantage of some of these features. First, edit the blabbyd.sh script to look like this:

#!/bin/sh
# blabbyd.sh: a blabby daemon, version 2
#
# note: variables now provided from environment with envdir:
#
#   .env/MESSAGE
#   .env/PAUSE
#
echo "starting blabbyd..."
# loop forever
while :
do
    echo "blabbyd: ${MESSAGE}"
    echo "blabbyd: sleeping for ${PAUSE} seconds..."
    sleep ${PAUSE}
done
# that's all, folks!

Note these differences from the first version:

1. The variables MESSAGE and PAUSE are no longer hard-wired into the script, but will instead be obtained from the environment.

2. The date stamps have been removed; these will now be provided by the logging service.

3. Output is simply left to stdout, rather than to an explicit LOG file; program output will then be captured by the daemontools logging service.

As before, install this script in /var/svc.d/blabbyd/blabbyd.sh and make it executable:

# cp blabbyd.sh  /var/svc.d/blabbyd/blabbyd.sh
# chmod 0755  /var/svc.d/blabbyd/blabbyd.sh

Then edit the run script script to look like this:

#!/bin/sh
# blabbyd/run
# run script for "blabby" daemon, version 2
exec 2>&1
echo "*** Starting service blabbyd..."
exec envdir ./env ./blabbyd.sh
# that's all, folks! 

As before, save the run script to /var/svc.d/blabbyd/run, and chmod 755.

In this script the first exec (line 4) is shell idiom for redirecting stderr to stdout. This is done so that output to both stderr and stdout streams may be captured by the logging service.

The second exec (line 6) calls the daemontools utility envdir with the arguments ./env and ./blabbyd.sh. The envdir utility runs blabbyd.sh with environmental variables set as defined in the directory ./env. Within the directory ./env, the name of the variable is given by the filename, and the value of the variable is provided by the file contents.

To illustrate, set some new parameters for the blabbyd.sh script:

# cd /var/svc.d/blabbyd
# echo 'Hi there!' > env/MESSAGE
# echo '7' > env/PAUSE

This will result in the blabbyd.sh script being run with the variable MESSAGE set to the value "Hi, there!", and the variable PAUSE set to value "7".

This is a classic idiom of the djb way: using the filesystem itself as a key/value pair data structure. In this case, the key is represented by the filename, and the value is the first line in the file. This allows djb tools to use the system's native file utilities to perform lookups, rather than developing extra parser code for each application.

Now to add the logging service. First create a subdirectory named log within the service directory:

# mkdir /var/svc.d/blabbyd/log

Create a run script for the logging service:

#!/bin/sh
# blabbyd/log/run
# run script for logging "blabby" daemon
exec multilog t ./main
# that's all, folks!

Save this to the file /var/svc.d/blabbyd/log/run, and chmod 755.

Notice that this script just runs the daemontools logging utility multilog. The t argument to multilog causes the output to be prepended with a TAI timestamp. The ./main argument is the directory where log files will be written and managed.

When the service is installed in /service via symlink, svscan will notice that /service/blabbyd also includes the directory /service/blabbyd/log. It will then automatically create a pipe from the blabbyd service to the logging service, so that any output from blabbyd.sh will become input to the multilog utility.

As blabbyd.sh periodically generates its noisy output to stdout, multilog will capture it, prepend a timestamp, and write it to a set of rotated log files maintained by multilog under log/main.

Install the service now to see all this in action:

# ln -s /var/svc.d/blabbyd  /service/blabbyd

Within a few seconds, svscan will notice the appearance of blabbyd and blabbyd/log in the /service directory. It will then start supervise on each of the run scripts, blabbyd/run and blabbyd/log/run, with a pipe between the two as described above.

Verify this with svstat:

# cd /service
# svstat blabbyd blabbyd/log
[XXX example output here]

Switch to another terminal and observe the log. We now use the -F option to follow the log file through rotations:

# tail -F /service/blabbyd/log/main/current

[On OpenBSD, use tail -f to follow the log through rotations.]

Every 7 seconds (as the parameter is now set in env/PAUSE) a new message line should be printed (as set in env/MESSAGE), preceeded by the unfamiliar string of characters that represent a TAI ("atomic") timestamp. TAI will be discussed in a later section; for now, view the TAI timestamp in a more readable notation by piping the output through the tai64nlocal utility:

# tail -F /service/blabbyd/log/main/current | tai64nlocal

As with the original version of blabbyd, you should observe what happens when the service is shut down (svc -d), brought up (svc -u), paused, continued, and so on. Experiment also with giving the service new parameters:

# echo "I'm hungry!" > /service/blabbyd/env/MESSAGE
# echo "13" > /service/blabbyd/env/PAUSE
# svc -t /service/blabbyd

This restarts blabbyd with new values for MESSAGE and PAUSE, as can be seen in the log output:

  [XXX Add log output!]

Cool! This is a common way of updating services under daemontools with new parameters on the fly.

Notice that we have been using the svc utility only for controlling and signaling the blabbyd service, while leaving the logging service alone. Although the logging service may also be controlled with all the svc commands, such as:

# svc -t /service/blabbyd/log

the general convention is to simply leave the log service running continuously. In that way no output from the primary daemon service will be lost.

Had enough? Removing the blabbyd service is similar to the first example:

# cd /service/blabbyd
# rm /service/blabbyd
# svc -dx . log

The last line brings down and exits both the blabbyd service (.) and its logging service (log) in the arguments passed to svc. Say goodbye to blabbyd!

Now that the blabby daemon is finally gone--and for the second time at that--we won't dare to bring her back. But there is still so much more to show of daemontools...

The djb way continues!