booting svscan

The standard daemontools installation procedure automatically installs the svscanboot command into your system's startup mechanism. For BSD systems, this is the line

/bin/csh -cf '/command/svscanboot &'

installed in /etc/rc.local, while in Linux/SystemV systems the line

SV:123456:respawn:/command/svscanboot

is installed in /etc/inittab.

The /command/svscanboot executable provided with daemontools is itself just a shell script that looks something like this:

#!/bin/sh
# WARNING: This file was auto-generated. Do not edit!

PATH=/command:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin

exec </dev/null
exec >/dev/null
exec 2>/dev/null

/command/svc -dx /service/* /service/*/log

env - PATH=$PATH svscan /service 2>&1 | \
env - PATH=$PATH readproctitle service errors: .....(400 dots total)

The script first sets the path that will be made available to all daemontools run scripts, blocks any input/output by redirecting standard streams to /dev/null, then shuts down any services that may already be running.

The script then starts svscan on the /service directory with a clean environment, redirecting stderr to stdout. Any output from svscan is then piped into the command listed on the last line of the script.

This brings us to the readproctitle command, and the central object of interest in the discussion that follows.

The purpose of readproctitle here is to give us some access to any error messages that may be generated by a running svscan process. The 400 dots that appear as the last argument to the command provide readproctitle with an in-memory buffer it uses to display whatever it reads from its standard input. It is then possible to view this display with a process status listing using the ps(1) command:

# ps -axww | grep readproctitle
[XXX sample output here]

The multiple ww option to ps will output the complete width of the line, rather than truncate to fit your terminal. If all is well, all you will see is the readproctitle command line with all those hundreds of dots.

When all is not well, however, the dots following readproctitle in the ps output will start filling up with the error messages from svscan, rolling in from the right. Let's say one of your run scripts is not set executable, a common error when you are adding a new service. Once every second, as supervise tries repeatedly to start the service, an error message will be generated by svscan and will start to scroll from right to left into the readproctitle log:

# ps -axww | grep readproctitle
[XXX sample output here]

That's the idea, anyway.

In practice, though, readproctitle has its downsides. For one thing, its behaviour is anomalous on the default installation of at least one of our reference platforms (FreeBSD 5.1). Other problems:

• output in ps listings is wrapped, truncated and hard to read
• no timestamps; when did the error happen?
• difficult to follow the log in real time, just ps over and over
• old error messages aren't cleared

But if we abandon readproctitle, how else are we going to know what's going on with svscan?

One answer is to simply use daemontools multilog instead.

Here's the svscanboot script from above, renamed to svscan-start, and adapted to use multilog in place of readproctitle:

#!/bin/sh
# svscan-start
# startup daemontools svscan with a multilog
#
# usage:  svscan-start [dir] [log]
#
# optional arguments:
#
#   dir:  directory to run svscan [/service]
#   log:  directory for multilog output [/var/multilog/svcan-${SERVICE_BASE}]
#
# ===
# WARNING: This file was not auto-generated.

## parameter setup:
DEFAULT_SERVICE=/service
SERVICE=${1:-${DEFAULT_SERVICE}}
SERVICE_BASE=`basename ${SERVICE}`
DEFAULT_LOG=/var/multilog/svscan-${SERVICE_BASE}
LOG=${2:-${DEFAULT_LOG}}

PATH=/command:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin

exec </dev/null
exec >/dev/null
exec 2>/dev/null

/command/svc -dx ${SERVICE}/* ${SERVICE}/*/log

{
  echo "*** Starting svscan on ${SERVICE}:"
  env - PATH=$PATH svscan ${SERVICE} 2>&1
} | \
env - PATH=$PATH setuidgid multilog multilog t ${LOG}

### that's all, folks!

Copy into /usr/local/bin/svscan-start and set executable, chmod 755. Then make the directory for the new multilog:

# mkdir -p /var/multilog/svscan-service
# chown multilog /var/multilog/svscan-service

Kill the existing svscan process, then reconfigure your system startup scripts. For BSD, edit the invocation in /etc/rc.local to read:

# daemontools svscan:
if [ -x /usr/local/bin/svscan-start ] ; then
    echo -n ' svscan'
    /command/pgrphack /usr/local/bin/svscan-start &
fi

For Linux/SystemV, edit /etc/inittab to:

SV:123456:respawn:/usr/local/bin/svscan-start

When svscan restarts, you should now see a nice timestamped multilog in /var/multilog/svscan-service:

# tail -f /var/multilog/svscan/current | tai64nlocal
[XXX sample output]

This gives immeasurably better feedback from svscan, especially helpful in trouble-shooting and problem-solving for the many services we will be installing along the djb way.

The svscan-start script shown here may also take optional arguments; see the notes in the script header. This allows a system to start svscan on additional service directories. For example, we sometimes setup a /service.test directory for the purpose of testing and debugging new services, before moving them into /service:

/usr/local/bin/svscan-start /service.test /var/multilog/svscan-test

What's with the /command/pgrphack instead of /bin/csh in the BSD startup? pgrphack accomplishes the same thing as the C shell here, which is to run svscan in a clean process state, in its own process group, and with a parent process ID of 1.

The difference is that the daemontools pgrphack utility is specifically designed to accomplish this task, and does it with an executable about one-tenth the size of the C shell.

As mentioned above, FreeBSD 5.x (release 5.1 in any case), has problems with readproctitle.

The default installation will show this in a process status listing with ps:

# ps -axww | grep readproctitle
397 con- S      0:00.61  (readproctitle)

Oops, where did the 400 dots go? All you will ever see is the (readproctitle) in parentheses. So much for getting any error reporting from svscan.

The problem arises because, for security reasons, FreeBSD no longer mounts procfs, the process file system, by default.

If you just gotta have it, procfs(5) can be mounted at system boot by adding the following line to /etc/fstab:

# Device  Mountpoint  FStype  Option  Dump  Pass#
proc      /proc       procfs  rw      0     0

When the filesystem is mounted, the full readproctitle display will return.

Be forewarned, however, that the maintainers of FreeBSD advise that the use of procfs is deprecated.

• Glenn Strauss provides a patch for daemontools to help log svscan output to multilog, for another way to bypass readproctitle. See http://www.gluelogic.com/code/daemontools/ for more information.