Voyager Processing at UMD

The daily processing pipeline at UMD downloads new mrt files from APL and runs python and bash scripts and compiled Fortran programs to update various plots and ascii database files available on the UMD Voyager web page. This document describes the pipeline as of February, 2024.

Table of contents

  1. History
  2. Directory Layout
  3. Requirements
  4. Running the pipeline
  5. overview
  6. Voyager mac web server


The pipeline was originally run on a DEC VAXstation(??) using compiled DEC Fortran code, IDL scripts and DCL batch files. When the VAXstation was decommissioned the pipeline and web pages were moved to a powerpc based Mac. Larry Bleau and Matt Hill ported the DEC Fortran code to compile with the g95 compiler from on a powerpc Mac, and ported the DCL scripts to bash scripts. When the pipeline was moved to an intel Mac mini the Fortran code was updated to work on little endian processors and to compile with the GNU gfortran compiler. The get_mrt bash script was replaced with the python script in order to add new features to the pipeline such as interactive plotting of fluxes and rates.

Directory layout at UMD


Running the pipeline

The pipeline is scheduled to run each day at 16:32 under the larrybleau account using the getmrt launchctl plist file. The pipeline can be started manually with the command

sudo launchctl start get_mrt

If the plist file is not loaded you can define the VOY and fky environment variables described above and run

${VOY}/bin/ new --filter

from the ${VOY} directory. overview

Run -h to get help on using the script. The script is run with the new and --filter option when started by launchctl

  1. If the command line argument "new" is used, the script downloads new zipped mrt files from APL to ${VOY}/mrt/zip/ using rsync. If a specific mrt file name int he format syyddd is used instead, rsync downloads the specified mrt file to the ${VOY}/mrt/zip/ directory if the file exists at APL. Login is handled by the pexpect python package. If no files are downloaded the script sends an email message and exits.

  2. Files that pass a simple Cruise5A check are unzipped in the ${VOY}/mrt/zip directory. The compiled c program reform is used to change the record format of the unzipped mrt files to one that can be read by our programs compiled with gfortran. The converted files are moved to the ${VOY}/mrt directory. If daily mrt files are being provided then old daily files with times covered by the new weekly mrt files are moved to ${VOY}/mrt/daily.

  3. The names of the new files are used to determine the year, first day and last day arguments used to run the ${VOY}/bin/filter_run bash script. filter_run runs the compiled Fortran program filter2 to extract valid mrt records from the new mrt files and save them to yearly mrt files for each spacecraft in the ${VOY}/filt directory.

  4. getmrt then runs the ${VOY}/bin/averages bash script to update the 1 day, 26 day, 52 day and yearly flux database files in ${VOY}/db. averages runs the ${VOY}/bin/voflx_avg bash script which runs the ${VOY}/bin/voflx_run bash script. voflx_run runs the compiled Fortran program ${VOY}/bin/voflx_cr5a to calculate the fluxes. voflx_avg then runs the ${VOY}/bin/vfmaster bash script. vfmaster runs the compiled Fortran program ${VOY}/bin/update_master to update the database files from the .ovf files created by voflx_cr5a.

  5. getmrt then runs the ${VOY}/bin/ python script to create the 26 and 52 day and yearly spectra plots. uses the ${VOY}/bin/ python script to make spectra plots for the given year and the ${VOY}/bin/ python script to update the 1 day, 26 day, 52 day and yearly time-intensity plots. All plots are available on the web page.

  6. Finally, updates the data used by the interactive plotting app and sends an email message with information about which mrt files were downloaded and the commands used to process them.

Voyager mac web server

The nginx web server is used to serve the Voyager web site at UMD. Most of the site is static html pages containing links to images and database files that are updated by the pipeline. A cron job runs the ${VOY}/bin/ python script on January 2 each year to update the spectra plot links in the html files for the new year.

The interactive plotting web app is created with Dash. It is run by the uwsgi python package and is started by launchctl.