Home
Latest News
Downloads
Repositories

Nbsp
Quick guide
Wiki
Article for Geo Quarterly

Npemwin
Project | Wiki
Manual
What's new

Support
Problem reports
Group
Contact







Filters

Standard filters

When all the frames that make up a file are received and the file is saved, the processor sends some information about the file to the installed filters, which can then process the file further. Several filters are included with the nbsp distribution, and to distinguish them from filters not supplied with the distribution we call them the standard filters. The emwin and gempak filters discussed above are two of them.

A few additional standard filters are the following.

rstfilter

The rstfilter processes the satellite image files and the text files, and saves a copy of them in a standard directory tree. In fact, this filter is invoked automatically by the emwin filter. Therefore, if you have enabled the emwin filter then the rst filter should not be enabled separately, or you end up duplicating its function. The intention of providing the rst filter separately is to permit its use even when the emwin server is not activated.

The rstfilter is enabled by setting

set feature(rstfilter) 1
in the features.conf file. The files must be installed in the configuration directory. They need not be edited unless the installation is not the default one, or if some special configuration is desired. A third file can be installed also (and edited if needed) but the presence of this file is not required to enable the filter.

The rstfilter can also generate radar gif images but this is not enabled by default. It requires the program ``gpmap_gif'' that comes with the gempak package. Thus, to enable the generation of radar images, the variable rstfilterrad_enable must be set to 1 in the file ``rstfilter.conf'', and the products for which the images are generated can also be specified in that file. The rst filter assumes that the radmap support files are stored in the ``/home/gempak/gempak/radmap'' directory, but a different location can be specified in the configuration files.

dafilter

This filter saves the files in a format that can be loaded directly by Digital Atmosphere (DA). The dafilter is enabled by setting

set feature(dafilter) 1
in the features.conf file, and the files must be installed in the configuration directory. The configuration file dafilter.conf is not required, but can be used to redefine some of the default parameters. The directory where the dafilter saves the files is by default ``/var/noaaport/data/digatmos'', but that can be redefined in its configuration file ``dafilter.conf''. In order to use the files from DA, the directory must be exported to the windows PC by samba.

panfilter

The panfilter (``PAN'' stands for Product Arrival Notification and it is a term borrowed from the Unisys prodman manual), is one mechanism for notifying another computer or computers when a particular file is received. The main motivation for enabling this in nbsp is to support a setup in which one or several computers wait for a message indicating that a particular product file has been received and saved in the spool directory by nbsp, so they can go and get it for further special processing. The nbsp computer can export by NFS, http or other means the spool directory, and the client computers can read just the file or request it by http or whatever have been set up.

There can be various form of notification, but at the moment the only one implemented is a UDP transmission. The panfilter has an rc file panfilter.rc where the rules for deciding what files to send and to whom are defined. When a given file is received and it matches a rule, the panfilter sends the client computer(s) a message containing the wmo header, the awips id if it is in the file, and the name of the file. With that information the client computer can construct the appropriate path or url to get the file and do what it wants with it.

The file panfilter.rc-ex contains more detailed explanations and examples, and the files ``nbspudpread.pl'' and ``nbspudpread.tcl'' (in the nbsp documentation directory ``/usr/local/share/doc'') are two simple udp reader examples.

The panfilter is enabled by setting

set feature(dafilter) 1
in the features.conf file.

metarfilter

MetarWeather is a nice program for Windows, free, available from www.nirsoft.net (in the ``Utilities'' section). The metar data from each station or collective is summarized and saved in a file in the ``metarweather'' subdirectory of the noaaport data directory and any of those files can simply be opened in MetarWeather (assuming that the noaaport data directory is exported via samba).

Then from the MetarWeather program, the data is obtained by opening the files that are in the subdirectory

metar/metarweather
If a collective file is opened (for example Louisiana/Louisiana.txt), then the last report from all the stations of the Lousiana collective are displayed. If a particular station file is opened, then the last reports from that station are displayed.

The metarfilter is enabled by setting

set feature(metarfilter) 1
in the features.conf file.

gribfilter

The gribfilter is enabled by setting

set feature(gribfilter) 1
in the features.conf file. The gribfilter is designed to work with GrADS, but it can be used also without it. In the latter case, the control files and the plots will not be generated, but all the data files will be collected and saved in
/var/noaaport/data/grib
In the default installation the gribfilter is enabled, and the GrADS related features are disabled. What the gribfilter does with the grib data files is explained in ``gribnames.README''.

When GrADS is installed the gribfilter has much more functionality. To use the gribfilter with GrADS, in the "gribfilter.conf" file (or in a copy of that file in the "site" subdirectory), the following settings must appear:

set gribfilter(gradsctlenable) 1; set gribfilter(gradsplotsenable) 1;
The additional features of the gribfilter with these settings are described in ``grads-grib.README''.

trackfilter

The trackfilter collects the data from the NHC bulletins and plots the tracks and forecasts of each hurricane. It is enabled by setting

set feature(trackfilter) 1
in the features.conf file. It uses GrADS to draw the plot and therefore that package must have been installed. If GrADS is not installed, the plot can be disabled by setting
set trackfilter(grads_enable) 0;
in trackfilter.conf. In thos case the filter only collects the data and saves in the data files in the /var/noaaport/data/track directory that can be used with other programs if desired.

The data and the plots are accesible from the built-in web interface.

msgfilter

The msgfilter is able to send email and sms (text) notifications based on product codes and geographical zones. The filter is enabled by performing the following steps:

The msgfilter is driven by an "rc" file that specifies who gets what and how. The "how" is specified by the transport mechanism and perhaps some flags. Currently, the only transport is smtp and the flags are "F" or "S" for sending the full file or a summary, respectively. In this case (smtp), the "who" is an email address.

What each subscriber gets is specified by a combination of product codes and zones using a regular expression syntax. For each incoming product, the filter scans the subscriber list and it builds up a list of recipients according to the regexp matching rules.

Once the filter has collected the recipients list(s), one for the F flag and another for the S flag, it composes the messages and hands off the delivery to the smtp server (configurable in "msgfilter.conf"). The msgfilter speaks smtp with the server, so that the server can be either the localhost (the default) or another computer; the filter doesn't care. In this way, the delivery is off to another process, in the same or a different computer, and the filter can go on minding its own business.

The list of "subscribers" is specified in the following form, one such term for each "subscriber", in the file "/usr/local/etc/npemwin/msgfilter.rc" (but see below):

lappend subscribers { transport://destination prodzone_spec flags }
Here prodzone_spec is a comma-separated list of terms where each term has the form
prod_code_regex-zone_code_regex
For example, given
lappend subscribers { smtp://nieves@noaaport.net (svs|tor)-txz14[5-9],svr-vaz014 flags }
then I would receive all svs and tor for TX zones 145 to 149, and also all SVR for VA zone 014. The comma separated list of product-zone combinations can be extended indefinitely.

The flags depend on the particular transport. For smtp it can be "F" or "S" to send the full file or the summary. Each transport defines its own set flags (for transports could be empty).

As already mentioned, the filter reads reads the file

/usr/local/etc/npemwin/msgfilter.rc
to get the list of subscribers. In practice the list should be written instead in the "site" file (the filter looks for it as well)
/usr/local/etc/npemwin/site/msgfilter.rc

The list can be kept in a separate flat file, in the format (space as separator)

transport://destination prodzone_spec flags
e.g.,
smtp://nieves@noaaport.net (svs|tor)-txz14[5-9],svr-vaz014 F; smtp://phonenumber@tms.suncom.com (svs|tor)-txz14[5-9] S;
If such a flat list is kept in the file
/usr/local/etc/npemwin/site/msgfilter-subscribers.txt
then it can be imported by "site/msgfilter.rc" file by writing in that file
msgfilter_add_flat_file "/usr/local/etc/npemwin/site/msgfilter-subscribers.txt"
The examples in tye files msgfilter.rc-ex and msgfilter-subscribers.txt-ex included in the distribution illustrate all this.

nbspfilter

The nbspfilter is a general purpose filter framework. As with the other supplied filters, it is controlled by a configuration file ``nbspfilter.conf'' and a run control (rc) script ``nbspfilter.rc''. Several options, including the location and name of the rc script, can be modified in its configuration file nbspfilter.conf, but that file is not required to be installed.

nbspfilter is written in tcl and, to enable it, the nbspfilter.rc script must be installed, populated with rules and actions, also in tcl.

nbspfilter is designed such that, after doing some preliminary work, it ``sources'' its run-control (rc) script nbspfilter.rc. The sample rc script nbspfilter.rc-sample provided with the distribution contains some background material and several examples illustrating how to write such a script and how it can be used.

User filters

We call user filters those not supplied with the program, either written by you or third-parties. There is no difference in functionality between standard and user filters, and this naming convention is only for reference.

Installing a user filter

After a filter is written, it is enabled in one of two ways. One is to modify the variable filterlist in the file ``nbspd.conf''. For example,

set filterlist "/usr/local/libexec/nbsp/gpfilter:\ /var/noaaport/filters/myfilter"
will enable the gempak filter, and a filter named myfilter in the directory /var/noaaport/filters. The second, recommended way, is to store the filter in the /var/noaaport/nbsp/dev directory. When the nbspd program starts, it will enable all the filters that it finds in that directory. The location of that directory can be modified by setting the devdir variable in nbspd.conf. If the nbspd program is already running, instead of stopping and restarting, you can send it a SIGHUP signal to reload all the installed filters in the dev directory.

Writing a user filter

A filter can be written in any language, e.g., perl or even C if you wish. When installed, the filter will receive in its standard input a one line text message for each received and processed file. The line is similar to this 

88164933 4 1 17 /var/noaaport/nbsp/spool/KMOB/kmob_sdus54-n0rmob.498745

The four numbers are the sbn sequence number, type, category and code of the product as defined by the noaaport system, which correspond to the variables seq, type, cat, code already mentioned in the nbspfilter description, and the rest is the full path of the saved file. It is now up to the filter to do whatever it wants with the file.

For example, to write in some file the sequence number and the file name for each received file, a perl script fragment would look as 

    if(open(OUT, ">/var/noaaport/list.log") == undef){
        syslog("err", "Could not open logfile. $!");
        exit;
    }
    
    while($line = <STDIN>){
        chop($line);
        ($seq, $type, $cat, $code, $fpath) = split(/\s+/, $line);
        $dirname = dirname($fpath);
        $fname = basename($fpath);
        print(OUT "$seq $fname\n");
    }

The only difference between writing the rc script of the nbspfilter vs writing such a stand-alone filter is that the nbspfilter has already done some of the work by identifying the various pieces of information about the saved file. Here it must be done scratch, but apart from that there is no difference, and of course the functionality of the nbspfilter (or any filter for that matter) can be imitated in perl and other languages.