SeedLink Design Notes and Configuration Tips

Andres Heinloo
August 31, 2000

Introduction

Seedlink is a system for near real time seismic data distribution. Like Comserv, it has a client-server architecture and uses the Mini-SEED data format. However, there are principal differences between the two systems:

Comserv's client-server interface is based on shared memory, so there is no way to run remote clients. Seedlink's client-server interface is based on TCP sockets, so clients can run on any system which has a TCP/IP connection to the Seedlink server (including the local PC).
In addition to data requests, Comserv supports mass recentering and calibration, file upload/download between the Quanterra and PC, setting various Quanterra parameters, status inquiries, etc., altogether 32 commands. Seedlink supports only data requests, because it does not make sense to send eg. ``mass recenter'' command from a remote host in such a distributed environment.

Seedlink protocol is very simple: data is sent in the form of 512-byte Mini-SEED packets with 8-byte Seedlink header. The header contains packet sequence number which is used to resume transmission where it left off: this makes possible to recover the connection in the case of network errors and also supports non-permanent connections (the ``dialup mode'').

Server side of the Seedlink protocol is implemented as a C++ module which is used as a basis of the Seedlink program (the Seedlink server). Client side is implemented as an extension to the Comserv package--most Comserv clients (netmon, datalog, datadump, msgmon, qplot, dpda, etc.) work unmodified in the Seedlink system.

Seedlink server uses only high-level socket interface, so it can run on any physical communication media which is supported by the underlying operating system; this includes modems, ISDN, and Ethernet among others. It is possible to link several Seedlink systems together--this allows both ``chain'' or ``star'' type communication models or a mixture of both.

Seedlink is designed to support unreliable connections--some number of recent packets are kept in the main memory for very efficient access, older packets are temporarily stored in the disk buffer. So it is possible to completely recover the connection even if the network is down for a long time. The size of memory and disk buffers can be specified by the user. No error correction is implemented by the Seedlink, however, because the TCP protocol guarantees error-free communication.

The data source of a Seedlink server can be anything which supported by a Seedlink ``plugin''. Data supplied by a plugin can be in a form of Mini-SEED packets or just raw samples. In the latter case, Seedlink server uses an integrated, flexibly configurable, Stream Processor module to create the datastreams and assemble Mini-SEED packets. Two plugins have been implemented--Comserv plugin is used to feed Comserv data into Seedlink and to create Seedlink ``chains''. Serial plugin is designed to support various serial digitizer protocols and can also read data from a UNIX fifo.

Seedlink and Comserv

The Seedlink program is a server, so we have to implement clients for data storage and visualization to make the system useful. On the other hand, the Comserv package has a full set of clients and it is widely used for VBB seismological data acquisition based on Quanterra dataloggers. By integrating Seedlink support into the Comserv server, we get a working system with minimal development and maintenance effort.

It is worth to note that the Comserv package used in the latest SeisComP distribution is completely backwards compatible with the Berkeley distribution. It is possible to compile the Comserv package with a regular C compiler on a Sun machine to get a big-endian version of the binaries with Seedlink support.

The drawback of this approach is that the design is not symmetric: Quanterra is a ``special case'' since it is directly supported by the Comserv while all other digitizers need some kind of Seedlink plugin for their support. To make the architecture more symmetric and easier to understand, we have to run two instances of the Seedlink server as seen below.

Example Configuration

In this paragraph, we are going to take a look how to set up an example Seedlink/Comserv system. Let's assume that we have six stations:

KRIS: local Quanterra;
EIL: remote Seedlink station;
EDD: local Earth Data digitizer;
ARB1
ARB2: arbitrary local data files;

We want to store the data of these stations locally and also (re)distribute the stations via a local Seedlink server. The configuration is shown on figure

. Note that Digiserv and Seedlink are two instances of the same program: one is used internally in the system, the other is used to serve outside world.

**Figure:** Data flow in a Seedlink/Comserv system
$\includegraphics{seedlink.eps}$

As a first step of setting up a Seedlink/Comserv system, we'll have to create user ``sysop'' and the /home/sysop directory containing subdirectories bin, cmds, config, data, digiserv, logs, and seedlink¹. All the binaries should be copied to the bin directory and sample config files (which will likely have to be modified) to the config directory. Other directories are used as follows:

data: is where station-specific configuration and Mini-SEED data is stored;
logs: is where log files and some Comserv stuff (*.lock, *.pid, and *.seq files) are located;
cmds: private directory of Netmon;
digiserv: private directory of Digiserv;
seedlink: private directory of Seedlink.

Setting Up serial_plugin

The configuration file for plugins is /home/sysop/config/plugins.ini. Section [edata] must be created here. The easiest way to do this is to take the file plugins.ini supplied in the distribution and set parameters to their right values. After this is done, the plugin can be tested by issuing a command in the bash shell:

    serial_plugin -v edata 255>data.dat

Type Ctrl-C to stop the program. The file data.dat should now contain some messages and data in Seedlink's internal format. If you want to test the digitizer without a GPS, you need to set pctime=1 in the configuration. If this works, we can go to the next step.

Setting Up digiserv

The digiserv program is actually just a soft link (shortcut) to seedlink which is used to make the system easier to understand and configure.

First we'll need to tell the Digiserv, how it should create Mini-SEED channels from the raw data supplied by the serial_plugin. This configuration is stored in streams.ini and the easiest way to set it up is to take streams.ini supplied in the distribution and make necessary modifications.

The configuration of Digiserv is stored in the [digiserv] section of the file seedlink.ini; sample seedlink.ini is supplied in the distribution.

Now we can test digiserv by entering at the shell prompt:

    digiserv -v

Along with Digiserv's messages, messages from serial_plugin should also appear, because it is started by the Digiserv. Let's open another xterm and type:

    telnet localhost 60999

If this succeeds then Digiserv is listening at this port. Now type:

    HELLO

Digiserv should respond with a message containing the version of the program and station name. Now enter:

    FETCH

Digiserv should respond with some garbage after a short time. If the garbage doesn't appear and the Serial Plugin was checked to be functional in the previous step, then there are most probably one or more of the following reasons:

station name in the [edata] section of plugins.ini doesn't match any in the [digiserv] section of seedlink.ini;
the station definition in the [digiserv] section of seedlink.ini doesn't contain proc= parameter which tells to use a stream processor;
stream tree's input names doesn't match plugin's channel names;
the root node of the stream tree definition doesn't lead to any streams;

The session can be closed by entering the BYE command. When you stop Digiserv (by typing Ctrl-C in the window where it was started), it will dump its memory buffers to +/home/sysop/digiserv/XXX.buffers+. This file can be analyzed with Mini-SEED tools to see if all the streams are there.

Let's hope that everything works, so we can go to the next step.

Setting Up the Comserv

Thorough description of the Comserv system can be found in the Comserv documentation (file comserv.pdf in the sources). Basically, you have a Comserv server for each station and several clients running. A Comserv client can talk to any combination of servers, but in most cases you have individual instances of clients for each station.

Most notable exception to this is the Netmon client: the job of Netmon is to query the servers for status of the clients and to start and shutdown servers and clients according to user requests. Before shutting down the system, Netmon first sends the ``suspend'' command to the servers, then waits until no client has pending data. This ensures that there will be no data loss when you restart the system: if you care about data gaps, you should use Netmon instead of just killing the programs.

When setting up a Comserv system, we will first have to create the files /etc/stations.ini and /etc/network.ini, which are located in /etc instead of /home/sysop/config for historical reasons. Former is a general Comserv file which is used by all Comserv programs to find station's individual config file. The server will also look for the source= parameter in this file to find out if it should connect a Seedlink server or a local Quanterra: you should set source=comlink for all your Quanterra stations and source=seedlink for all remote Seedlink and local Digiserv stations.

/etc/network.ini contains system-wide configuration of the Netmon client. The content of this file is thoroughly documented in the file netmon.txt. Sample versions of stations.ini and network.ini can be found in the distribution.

Each station has an config file, called station.ini, which is located in that station's directory. Comserv server uses the [comlink] or [seedlink] section of this file, depending on the setting of the source= parameter in /etc/stations.ini. Traditionally, the station.ini file has a section for each Comserv client and Netmon will warn if it is asked to run a client which has no section in station.ini. Some clients (normally not run by the Netmon) have separate config files, though.

Once /etc/stations.ini and /etc/network.ini have been set up, a separate directory, as specified in the /etc/stations.ini, has to be created for each station, containing the station.ini file. You can replicate the version which is found in the distribution, change station codes to their right values and make other necessary modifications. Be sure to set the segid= parameter to a unique value for each station--it will save you lots of head scratching later. You can delete the [comlink] section if the station is a remote Seedlink station or a local station served by the Digiserv. In the same way, you can delete the [seedlink] section if the station is a local Quanterra station.

For Seedlink stations, you should set the port= parameter to the Internet address of the Seedlink server; for Digiserv stations (which are technically local Seedlink stations), you should set the port= parameter to something like localhost:60999, depending on the port setting in the [digiserv] section of seedlink.ini. The uptime=, standby=, schedule=, ifup=, and fdown= parameters should be commented out, unless you want to run the Comserv in the ``dialup mode''².

One more parameter which deserves attention is netto= (network timeout). If the Comserv server doesn't receive any data from Seedlink in this number of seconds, it closes the connection and opens anew. It doesn't make sense to set netto= to a value which is smaller than the dialing interval if you have a dialup link somewhere in the way, unless you want to get lots of messages in your log files.

Here are some very important things to note about Comserv's clients:

Each client in the Comserv system in identified by a four-letter name, and it is important to use this name in the configuration, eg. in the [netm] section in station.ini, you should use

    client1=dlog,/home/sysop/bin/datalog

and not try to use eg.

    client1=datalog,/home/sysop/bin/datalog

because Netmon queries the servers for running clients, and it will not notice that Datalog is running unless you use the right client name. The client names which might be of interest are as follows:

DLOG: datalog
TAPE: datadump
SLNK: comserv_plugin
QPLT: qplot
MSGM: msgmon
NETM: netmon

A Comserv's client can run in one of the following modes: reserved, foreign, blocking, transient. Sometimes the clients are called ``reserved clients'', ``blocking clients'' and so on, but the mode is not really a property of a client, but depends on the server settings. The point is that only blocking clients can receive data reliably. On the other hand, blocking clients have to run permanently, because they are ``unblocked'' if they don't bother the server during the timeout period. It doesn't help to set the timeout to a very large value, because when the buffers of the Comserv server are full, it will block the whole system, waiting for blocking clients to fetch the data. This all implies that clients which are not started at bootup and running all the time cannot receive data reliably; this can be considered a Comserv design flaw.

This issue is not important if you use a local Quanterra--in this case the data is coming in at a determined slow speed, and the clients have no problems to fetch it all. But in the case of Seedlink, the data can come in large chunks after the net has been down for some time (or some node is operating in the dialup mode). In this case, there will be lots of tiny data gaps if clients are not running in the blocking mode.

The bottom line is that Datalog and Comserv Plugin should always run in the blocking mode. If you are using these programs, write into your station.ini files:

    client1=dlog,600
    client2=slnk,600

On the other hand, Datadump should never run in the blocking mode, even if it is permanently running, because Datadump does not request any data from the Comserv server. The only reason for implementing Datadump as a Comserv client was probably to make it monitored by the Netmon.

Further down in the station.ini file, you will find Datadump's section which I am not going to explain here, because I don't know much about it, and Netmon's and Datalog's sections which I am not going to explain here either, because good documentation exists in the files netmon.txt and datalog.txt.

When the Comserv has been set up, it's time to test the system on the next level. First run the Digiserv as follows:

    digiserv -v > digiserv.log 2>&1 &

Now start Comserv server for each of the stations, eg.:

    server KRIS > KRIS.server.log 2>&1 &
    server EIL > EIL.server.log 2>&1 &
    server XXX > XXX.server.log 2>&1 &

In this point, check the logfiles for possible error messages. If everything seems normal, try to run the dataread client--it will attach to all available stations and output a message for each packet received. Stop the Dataread by typing Ctrl-C, and the servers by

    killall server

Now type

    netmon -D -B -l -b

to start the Comserv system in it's full. The startup can be rather slow, depending on the parameters in /etc/network.ini. You can type, eg.

    ps axu | grep sysop

periodically to see when all the processes have been started. The logfiles are written in the /home/sysop/logs directory. The Comserv system should be now fully functional and Mini-SEED data should appear in the subdirectories under the /home/sysop/data/ directory. You can shut down the Comserv by typing:

    netmon -t all

Note that this leaves the Netmon daemon running, so next time you should use ``netmon -s all'' instead of ``netmon -D -B -l -b'' unless you have previously killed the Netmon daemon (or rebooted the PC).

Setting Up the Seedlink server

The configuration of the Seedlink server is located in the [seedlink] section of the file seedlink.ini, and the copy provided in the distribution is rather heavily commented. All you have to do is tweak the configuration if necessary and start seedlink by typing:

    seedlink -v > /home/sysop/logs/seedlink.log 2>&1 &

Because Seedlink is the same program as Digiserv, you can test it the same way you tested Digiserv before (eg. telnet, HELLO, FETCH, ...). Further testing can be performed by setting up a remote Comserv system.

Note that you can reboot the Seedlink server without any data loss--it resumes where it left off when it comes up. You should not stop the Seedlink for a longer time than the timeout of the Comserv's SLNK client, though.

Moreover, Comserv itself can be rebooted without data loss as well if you use Netmon³. In the case of Quanterra stations, the Quanterra keeps track which packets have been acknowledged by the Comserv server. In the case of Seedlink and Digiserv stations, the Comserv server writes the sequence number to a file like XXX.seq in the /home/comserv/logs directory upon exit and resumes from the next Seedlink packet when it is started. If you don't want the Comserv server request any old data, you can remove the corresponding *.seq file before you start Comserv.

So if you are using only Seedlink and Comserv stations, you can safely reboot your PC. Quanterra comlink buffers are rather small, so you should restart the system as soon as possible. Seedlink doesn't have this problem, because it uses the disk buffer which is normally rather large.

The disk buffer of the Seedlink is usually physically located in the /home/sysop/seedlink directory, and it can be configured by the segments= and segsize= parameters. For efficiency reasons, Seedlink does not search any data from the disk, but instead keeps internal data structures, so it ``knows'' where the data is located. For this reason, you should not modify the disk buffer by hand--Seedlink gets very confused if it wants to read some data and it isn't there. All you can do is to remove all of the files of a station when the Seedlink is not running (eg. rm XXX.* if you have station XXX) to reset the Seedlink to its initial state. When you stop Seedlink, it writes *.buffers and *.segments files; if Seedlink is not running and there are no *.segments files, then the disk buffer is abandoned and you should remove other files as well. Note that Seedlink removes *.buffers and *.segments files itself when it starts.

As mentioned above, Seedlink does not search any data from the disk, so it makes not sense to create files in the disk buffer by hand and hope that Seedlink will read them. If you want to feed arbitrary data to Seedlink, you'll have to implement a plugin. The plugin interface is very simple, just take a look at the file plugin.h in the Seedlink sources!

Now you can start the complete system like this

    digiserv -v > /home/sysop/logs/digiserv.log 2>&1 &
    seedlink -v > /home/sysop/logs/seedlink.log 2>&1 &
    netmon -D -B -l -b

and to shut down the system, type:

    netmon -t all
    killall seedlink
    killall digiserv

Note that after typing ``netmon -t all'' you should wait a few seconds to ensure that Seedlink has got all pending data from the Comserv. SeisComP distribution includes a script check_seiscomp which can be used to properly start up and shut down the system. If you would like to install this script as a cron job to check the system every 10 minutes, enter this at your shell prompt:

    echo "*/10 * * * * /home/sysop/bin/check_seiscomp check" | crontab -

Advanced Issues

Syslog

If you wish to send messages to syslog, you can use Digiserv and Seedlink without the ``-v'' option and remove ``-v'' from all the plugins in the seedlink.ini as well. Using Syslog has mainly two advantages: it can sort messages according to severity level, and supports automatic removal of old messages (using logrotate etc.).

The ``local0'' facility is used with four levels as follows:

err: non-recoverable errors--usually the Seedlink will exit if such an error occurs;
warning: recoverable, but important error conditions;
notice: normal, but significant messages;
info: startup messages and other stuff.

For example, if you wish to send messages of levels ``err'' and ``warning'' to seedlink-warn.log and messages of levels ``notice'' and ``info'' to seedlink-info.log, you can add these two lines into your /etc/syslog.conf file:

    local0.info;local0.!warning    /home/sysop/logs/seedlink-info.log
    local0.warning                 /home/sysop/logs/seedlink-warn.log

In this case you would normally want to disable logging Seedlink messages to other logfiles, so you should add ``local0.none'' to other selector fields. Please take a look at syslogd(8) and syslog.conf(5) manpages for more information.

When using Syslog, you can start the system in a bit simpler way:

    digiserv
    seedlink
    netmon -D -B -l -b

TCP Wrapper

The TCP Wrapper Library implements host access control and security checks. Please refer to hosts_access(5), hosts_options(5), and tcpd(8) manpages for details. Seedlink uses the TCP Wrapper by default. For example, to disallow outside connections to Digiserv, add this line to /etc/hosts.deny:

    digiserv : ALL EXCEPT localhost

Footnotes

¹SeisComP system usually includes Station Operation Manager and Data Request Manager which have their own directories, but setting up these components is not explained in this document.

²Dialup mode is only useful in the case when the remote station is accessible by a communication line where the costs are relational to the connection time and not to the number of bytes transmitted (eg. phone line or ISDN). In this case, the Comserv server will not keep the connection permanently open, but rather opens the connection periodically for a short time to download all data available.

³In this case, Comserv should be shut down before Seedlink, because otherwise the Comserv server will have