Madrigal Database Installation/Updating/Admin

Introduction

This document describes how to install or update a Madrigal installation, how to add new data to Madrigal, and some administration issues. The Madrigal installation has been tested under Red Hat Linux on a Pentium PC and SunOS 5.7 on several SPARC workstations and servers. The Gnu installation also works on Sun systems with minor changes in library locations and compiler options.

To create a new installation of Madrigal

Create a directory to be used as your Madrigal root directory on your solaris or linux server. This directory will be refered to as MADROOT, and the full path must be less than 40 characters.
Create the environment variable MADROOT with that directory path.
Download the Madrigal distribution file, madrigal.tar.Z to MADROOT.
uncompress madrigal.tar.Z
tar -xf madrigal.tar
Repeat the 3 steps above with experiments.tar.Z and geofil.tar.Z. You may also add you own experiments at this time. See section on adding experiments to madrigal.
Create two virtual directories on your webserver for your madrigal cgi files (which must allow scripts to run) and for your madrigal html files.
In madroot, there will now be a file called madrigal.cfg.template. Copy it to madrigal.cfg, and edit all the configuration parameters, as described in the following section.
Edit the file MADROOT/metadata/siteTab.txt with a unique id to include your site. If you do not want all the sites in siteTab.txt included in your combined inventory listing, just remove these sites from siteTab.txt. Links to data from sites you leave in the siteTab.txt file will appear in your Madrigal installation.
Be sure to cd to MADROOT before running the following step. Then you should be able to complete the installation simply by typing
```
    sh doc/installMadrigal.gnu $MADROOT TCLSH
```
or
```
    sh doc/installMadrigal.solaris $MADROOT TCLSH
```
where TCLSH is a tcl interpreter on your system, probably just "tclsh" if the interpreter is in your execution path, and "gnu" or "solaris" is selected depending on your compiler. There may be a long pause when running updateMaster near the end of the installation since the instParmTab.txt metadata file is being built for the first time by examining every data file, but future calls to updateMaster will be much faster since only new experiments are examined. Help with any installation errors is available from openmadrigal-developers@openmadrigal.org.
If there were no errors, your madrigal installation should be running at the url given by MADSERVERROOT. To configure two new features of Madrigal, private files and user-added notes, see the section on configuring optional Madrigal parameters.
If you own Matlab and you want to interface to Madrigal via the Madrigal Matlab API methods, you need to configure Matlab as described in the Madrigal Matlab API documentation.
If you want to receive notices about updates to Madrigal, sign up for the openmadrigal-admin mailing list under www.openmadrigal.org.
If you want to make your data available via other Madrigal sites, email the new line you added to the siteTab.txt file to openmadrigal-admin@openmadrigal.org. By adding this line to the existing siteTab.txt file, links to your data will appear at other Madrigal sites.

Editing the madrigal.cfg file

The madrigal.cfg file contains all the configuration information specific to your installation. This section discusses how to edit that file. Note that if you are updating Madrigal, check the new version of madrigal.cfg.template to see if it contains any new parameters that might not appear in your existing madrigal.cfg file.

To compile the Madrigal libraries and executable programs, you will need a C compiler and a FORTRAN 77 compiler. Makefiles are provided for Sun Solaris compilers and GNU compilers. The GNU compilers may be downloaded from the GNU Website.You only need to edit the compiler/make paths for the platform you are using; the others are ignored

By default, during the installation python will be installed on your server, along with the module PyXml, under MADROOT/bin and MADROOT/lib. If you do not want a local version of python installed, you can edit the madrigal.cfg file to uncomment the PYTHONEXE line. Check with the release notes to insure that your version of python is acceptable. The PYTHONEXE line must be the full path to your pre-existing python executable. Also, in that case you must manually install the latest version of the PyXml module on this version of Python. PyXml is available at http://sourceforge.net/projects/pyxml.

General information on python can be found on the Python Website.

The other prerequisite is Tcl/Tk 8.0 or later, which may be downloaded from the Tcl Resource Center.

Site-specific Specifications in madrigal.cfg

MADROOT - Madrigal root directory (absolute). This must be set as an environmental variable.
MAKESOLARIS - Solaris make program
CCSOLARIS - Solaris C compiler
FCSOLARIS - Solaris Fortran compiler
MAKEGNU - Gnu make program
CCGNU - Gnu C compiler
FCGNU - Gnu Fortran compiler
FORTLIBSOLARIS - Solaris Fortran library directory (where libF77*, etc., are)
TCLLIBDIR - Root directory in which Tcl was installed
TKLIBDIR - Root directory in which Tk was installed
TCLLIB - Location of the Tcl library
TKLIB - Location of the Tk library
TCLINCLUDE - Location of the Tcl include files
TKINCLUDE - Location of the Tk include files
TCLSH - Tcl interpreter
TCLLIBPATH - Tcl Library path. If you already have a directory where Tcl packages are stored, you can use that. Otherwise, the default, MADROOT/madtcllib, is OK.
MADSERVER - Web server for accessing Madrigal
PYTHONEXE - Optional - set to the full path to the python exe. This is required only if you do not want an internal version of python installed (the default)
MADSERVERROOT - Document directory relative to MADSERVER. This directory should not allow files to be executed.
madrigal/cgi-bin - CGI script directory relative to MADSERVER. The web server will need write permission in this directory. This directory should be set to execute only.
MADSERVERSERVLET - Servlet directory relative to MADSERVER. Not used at present, so just set it to MADSERVERROOT/servlets
MADSERVERDOCABS - Directory (absolute) where Madrigal documentation and images necessary for the Web software should be installed.
madrigal/cgi-binABS - Web server's CGI directory
MADSERVERSERVLETABS - Web server's servlet directory. Again, not used at present - just set to $MADROOT/servlets
SITEID - Site ID - Must be unique and same as in siteTab.txt
HTMLSTYLE - Body style of html pages
INDEXHEAD - Heading in the top-level Madrigal page
CONTACT - Mailto link of contact person(s) for this madrigal installation. Multiple email addresses may be included if separated by commas.
MAILSERVER - Name of mailserver (possibly localhost if running sendmail)
NOTESMANAGER - If you want someone to be notified whenever a user adds a note to an experiment, uncomment this optional line and add the email address. See below for a discussion of the optional notes feature and how to configure it after installation.
MAXGLOBALQUERIES - The maximum number of global queries you want to allow to run on your webserver at any one time. Since a global query can take minutes or even hours to run, setting this value will limit the number of these background jobs running on your webserver. Users who request a global query when the server is at this maximum already will get a message requesting them to resubmit the query later. A future release of Madrigal will queue these overflow queries for later running rather than rejecting them.

Configuring new Madrigal features

The following two features of Madrigal were added by Tony Van Eyken. They are configured as follows:

User-added notes

This new feature of Madrigal allows users to append notes from the madExperiments search results page. This feature is only enabled if the web-server has write permission in the particular MADROOT/experiments directory where the experiment data is located. To allow this feature for all data, set all directory permissions below MADROOT/experiments to be writable by the web server. To disallow this feature for all data, set all directory permissions below MADROOT/experiments to not be writable by the web server.

As mention under editing madrigal.cfg, the NOTESMANAGER parameter allows a user to be notified each time a note is added. This parameter can be commented out if no notification is required.

Restricted data files

This new feature of Madrigal allows data files to be made either publically available, or restricted to a limited set of IP address listed in a file.

To disable this feature and make all your data public, run the following from the $MADROOT directory:

 $MADROOT/bin/setAccess $MADROOT/experiments public

To enable this feature, you must first create a file called "trustedIPs.txt" under MADROOT. This file should contain the IP addresses of hosts considered to be part of the private group, or partial IPs if whole sub-nets should be included. This file is not created during installation. Anyone whose browser's IP matches this list will be able to view files marked as private. For example, if trustedIPs.txt contains the line 132.197.*, any IP address that begins 132.197 will have private access.

Also, if the fileTab.txt file for that experiment can be overwritten by the web server, anyone whose browser's IP matches this list will be able to to change the access of any file between public and private from the madExperiments listing page.

To change a large number of experiments to be either public or private, run the following script from the $MADROOT directory:

$MADROOT/bin/setAccess dirPath [public || private]

where dirPath is the full path name of any directory in $MADROOT/experiments, and the second argument is either public or private. This script will set all experiments in dirPath and below to be public or private.

The access permission for any given file is set in the fileTab.txt file. See metadata documentation for details.

To update an existing installation of Madrigal

Back up all files in MADROOT/metadata. These backups will only be needed if you have modified siteTab.txt, typeTab.txt, or instTab.txt, and not updated Madrigal cvs with your changes. It is important to keep these files consistent between all Madrigal sites that share links to data, so please notify us of any changes.
Download the Madrigal distribution file, madrigal.tar.Z to your MADROOT directory.
uncompress madrigal.tar.Z
tar -xf madrigal.tar (This will not overwrite any file meant to be modified by a specific Madrigal installation.)
Compare the new version of madrigal.cfg.template file to your existing madrigal.cfg file found under the MADROOT directory. If any new parameters have been added to the end of the file, add them to you madrigal.cfg file and edit just those entries as described above.
Diff the three backed-up metadata files mentioned in the first step with the installed versions in MADROOT/metadata, and make any changes needed.
Be sure to cd to MADROOT before running the following step. Then you should be able to complete the update simply by typing
```
    sh doc/installMadrigal.gnu $MADROOT TCLSH
```
or
```
    sh doc/installMadrigal.solaris $MADROOT TCLSH
```
where TCLSH is a tcl interpreter on your system, probably just "tclsh" if the interpreter is in your execution path, and "gnu" or "solaris" is selected depending on your compiler. Help with any installation errors is available from openmadrigal-developers@openmadrigal.org.
If there were no errors, your madrigal installation should be running at the url given by MADSERVERROOT. To configure two new features of Madrigal, private files and user-added notes, see the section on configuring optional Madrigal parameters.

To add/modify data to Madrigal

There are three separate cases to deal with when adding data to Madrigal:

Updating metadata files, or links to other sites.
Coping entire experiment directories from one Madrigal site to another.
Adding entirely new data files.

Updating metadata files, or links to other sites.

If you manually modify a metadata file in $MADROOT/experiments, or if you simply want to update your links to experiments at other Madrigal sites listed in $MADROOT/metadata/siteTab.txt, run:

cd $MADROOT
$MADROOT/bin/updateMaster

Copying entire experiment directories from one Madrigal site to another.

If you want to copy data from one Madrigal site into yours (such as is done during the standard installation with the test data from Haystack) copy all the desired data from another Madrigal site into the appropriate directory in $MADROOT/experiments. Then follow these steps:

cd $MADROOT
tclsh configureExperiments
$MADROOT/bin/updateMaster

The script configureExperiments automatically edits the metadata files, changing the site id to your site id.

Adding entirely new experiments or data files.

The present release of Madrigal requires that new experiments be put in a directory that follows a directory path convention. This convention is the following:

 
$MADROOT/experiments/<4 digit year>/<3 letter inst mnemonic>/<03sep97 type date>
Example:  /opt/madrigal/experiments/1998/mlh/03sep97

After you have created a Madrigal or Cedar format data file that you want to put in Madrigal, you need to create the metadata files that go in the same directory as that new file. This can be done by copying files from other directories and editing them by hand, or by using the script called genExp that will automatically generate those files. The script genExp can also be used to add new data files to an existing experiment.

To use genExp, put your new data files in some directory outside of $MADROOT/experiments. Then create a text file with seven whitespace separated columns, and one row for each new file. These columns are:

filename (example: /usr/brideout/dst570101g.001)
relative exp dir to $MADROOT/experiments (example: 1957/dst/01jan57)
instrument code (example: 212)
experiment name (example: "DST index")
file category (1=default, 2=variant, 3=history)
action (1 to create an entirely new experiment, 3 to add data file(s) to existing experiments)
description of file status: preliminary, final. Use quotes to add text with spaces.

If creating a new experiment, the genExp scripts will create the proper directories in $MADROOT/experiments, copy the data files in, and then create the needed metadata files. If just adding a new data file, the genExp scripts will copy the data files in, and then modify the file fileTab.txt. Note that (for the moment) you will need to manually edit the file fileTab.txt to modify the status of existing data files, such as changing status from default to history. The updateMaster script collects data from individual experiment metadata files and updates the summary metadata files in $MADROOT/metadata.

When the text file is ready, run the following three steps:

cd $MADROOT
$MADROOT/bin/genExp (name of text file)
$MADROOT/bin/updateMaster

Sharing data between Madrigal sites

When users look for data on your Madrigal site, they may also search for data at other Madrigal sites. To make this occur quickly, the updateMaster script downloads the experiment lists from all sites listed in the $MADROOT/metadata/siteTab.txt file. If you want to keep this data up to date, it is recommended that you set $MADROOT/bin/updateMaster to run on a regular biases via a cron job. Typically, updateMaster will run in under 2 minutes.

Graceful shutdown

Since this release of Madrigal includes the ability to generate long-running global searches of the database, it is possible for queries to be killed when rebooting the server. This can be avoided when there is a need to shut down the server by setting MAXGLOBALQUERIES to 0 in the madrigal.cfg file (this will disallow any new global queries - users will receive a message that the server is busy). When the file MADROOT/metadata/userdata/queryPid.txt, which lists the pid's of all running global queries, disappears, all queries are complete, and it is safe to reboot. Be sure to change MAXGLOBALQUERIES after rebooting to reenable global queries.

Known issues

Other compilers

If you are using compilers other than Solaris or GNU, use one of the supplied makefiles as a template. The only section of the makefile which probably will need to be changed is the compiler flag specification, since these are not standard between compilers. You may want to change the compiler flags even if you are using Solaris or GNU compilers, for example, to include debugging options.

Revised: March 6, 2004