Voyager Processing at UMD

The daily processing pipeline at UMD downloads new mrt files from APL and runs python and bash scripts and compiled Fortran programs to update various plots and ascii database files available on the UMD Voyager web page. This document describes the pipeline as of February, 2024.

Table of contents

1. History
2. Directory Layout
3. Requirements
4. Running the pipeline
5. getmrt.py overview
6. Voyager mac web server

History

The pipeline was originally run on a DEC VAXstation(??) using compiled DEC Fortran code, IDL scripts and DCL batch files. When the VAXstation was decommissioned the pipeline and web pages were moved to a powerpc based Mac. Larry Bleau and Matt Hill ported the DEC Fortran code to compile with the g95 compiler from g95.org on a powerpc Mac, and ported the DCL scripts to bash scripts. When the pipeline was moved to an intel Mac mini the Fortran code was updated to work on little endian processors and to compile with the GNU gfortran compiler. The get_mrt bash script was replaced with the getmrt.py python script in order to add new features to the pipeline such as interactive plotting of fluxes and rates.

Directory layout at UMD

• /Users/larrybleau/Voyager - top level of the directory hierarchy
  • bin - contains compiled Fortran code, bash scripts and python scripts
  • data - contains the .ivf files used by voflx_cr5a
  • db - contains the flux database files
  • filt - contains yearly filtered mrt files
  • idl - contains the idl scripts used for plotting before the python plotting scripts were created
  • log - contains log files output during processing
  • mrt - contains the processed mrt files suitable for use by the compiled programs
    • daily - daily mrt files were moved here when the weekly mrt files covering the same time period were downloaded
    • zip - contains the zipped mrt files downloaded from APL
  • prog - contains powerpc versions of the Fortran code
  • prog_i386 - contains intel versions of the Fortran code. Note that the compiled programs are 64 bit in spite of the name of this directory
  • work - contains .dat, .ovf, .prt and .spike files generated during processing
  • www - symlink to the web root directory

Requirements

• The environment variable VOY must be set to the top level directory of the pipeline directory hierarchy before starting the pipeline. At UMD it is set to /Users/larrybleau/Voyager
• The environment variable fky must be set to the cryptographic key used to decode the password rsync uses to log in to APL.
• IDL: we use IDL version 8.4
• Python: we use python 3.12 though the python scripts will probably work with python 3.8 and higher. The following python packages are required. The list shows the versions we are using as of February, 2024.
  • cryptography: version 42.0.3
  • numpy: version 1.26.4
  • pandas: version 2.2.0
  • pexpect: version 4.9.0
  • Pillow: version 10.2.0
  • dash: version 2.15.0, used for the interactive plotting web app
  • uwsgi: version 2.0.24, used for the interactive plotting web app
• Fortran compiler: we currently use gfortran 13.2 from gcc.gnu.org
• C compiler: we use the clang compiler included with Xcode

Running the pipeline

The pipeline is scheduled to run each day at 16:32 under the larrybleau account using the getmrt launchctl plist file. The pipeline can be started manually with the command

sudo launchctl start get_mrt

If the plist file is not loaded you can define the VOY and fky environment variables described above and run

${VOY}/bin/getmrt.py new --filter

from the ${VOY} directory.

getmrt.py overview

Run getmrt.py -h to get help on using the script. The script is run with the new and --filter option when started by launchctl

1. If the command line argument "new" is used, the getmrt.py script downloads new zipped mrt files from APL to ${VOY}/mrt/zip/ using rsync. If a specific mrt file name int he format syyddd is used instead, rsync downloads the specified mrt file to the ${VOY}/mrt/zip/ directory if the file exists at APL. Login is handled by the pexpect python package. If no files are downloaded the script sends an email message and exits.

2. Files that pass a simple Cruise5A check are unzipped in the ${VOY}/mrt/zip directory. The compiled c program reform is used to change the record format of the unzipped mrt files to one that can be read by our programs compiled with gfortran. The converted files are moved to the ${VOY}/mrt directory. If daily mrt files are being provided then old daily files with times covered by the new weekly mrt files are moved to ${VOY}/mrt/daily.

3. The names of the new files are used to determine the year, first day and last day arguments used to run the ${VOY}/bin/filter_run bash script. filter_run runs the compiled Fortran program filter2 to extract valid mrt records from the new mrt files and save them to yearly mrt files for each spacecraft in the ${VOY}/filt directory.

4. getmrt then runs the ${VOY}/bin/averages bash script to update the 1 day, 26 day, 52 day and yearly flux database files in ${VOY}/db. averages runs the ${VOY}/bin/voflx_avg bash script which runs the ${VOY}/bin/voflx_run bash script. voflx_run runs the compiled Fortran program ${VOY}/bin/voflx_cr5a to calculate the fluxes. voflx_avg then runs the ${VOY}/bin/vfmaster bash script. vfmaster runs the compiled Fortran program ${VOY}/bin/update_master to update the database files from the .ovf files created by voflx_cr5a.

5. getmrt then runs the ${VOY}/bin/webplot.py python script to create the 26 and 52 day and yearly spectra plots. webplot.py uses the ${VOY}/bin/spectraplotter.py python script to make spectra plots for the given year and the ${VOY}/bin/tiplotter.py python script to update the 1 day, 26 day, 52 day and yearly time-intensity plots. All plots are available on the voyager-mac.umd.edu web page.

6. Finally, getmrt.py updates the data used by the interactive plotting app and sends an email message with information about which mrt files were downloaded and the commands used to process them.

Voyager mac web server

The nginx web server is used to serve the Voyager web site at UMD. Most of the site is static html pages containing links to images and database files that are updated by the pipeline. A cron job runs the ${VOY}/bin/Update_Web_Pages.py python script on January 2 each year to update the spectra plot links in the html files for the new year.

The interactive plotting web app is created with Dash. It is run by the uwsgi python package and is started by launchctl.