Voyager Processing at UMD

The daily processing pipeline at UMD downloads new mrt files from APL and runs python, bash and IDL scripts and compiled Fortran programs to update various plots and ascii database files available on the UMD Voyager web page. This document describes the pipeline as of May, 2020.

Table of contents

1. History
2. Directory Layout
3. Requirements
4. Running the pipeline
5. getmrt.py overview

History

The pipeline was originally run on a DEC VAXstation(??) using compiled DEC Fortran code, IDL scripts and DCL batch files. When the VAXstation was decommissioned the pipeline and web pages were moved to a powerpc based Mac. Larry Bleau and Matt Hill ported the DEC Fortran code to compile with the g95 compiler from g95.org on a powerpc Mac, and ported the DCL scripts to bash scripts. When the pipeline was moved to an intel Mac mini the Fortran code was updated to work on little endian processors and to compile with the GNU gfortran compiler. The get_mrt bash script was replaced with the getmrt.py python script in order to add new features to the pipeline such as interactive plotting of fluxes and rates.

Directory layout at UMD

• Voyager - top level of the directory hierarchy
  • bin - contains compiled Fortran code, bash scripts and python scripts
  • data - contains the .ivf files used by voflx_cr5a
  • db - contains the flux database files
  • filt - contains yearly filtered mrt files
  • idl - contains the idl scripts used for plotting
  • log - contains log files output during processing
  • mrt - contains the processed mrt files suitable for use by the compiled programs
    • daily - daily mrt files were moved here when the weekly mrt files covering the same time period were downloaded
    • zip - contains the zipped mrt files downloaded from APL
  • prog - contains powerpc versions of the Fortran code
  • prog_i386 - contains intel versions of the Fortran code. Note that the compiled programs are 64 bit in spite of the name of this directory
  • work - contains .dat, .ovf, .prt and .spike files generated during processing
  • www - symlink to the web root directory

Requirements

• The environment variable VOY must be set to the top level directory of the pipeline directory hierarchy before starting the pipeline.
• The environment variable fky must be set to the cryptographic key rsync uses.
• IDL: we use IDL version 8.4
• Python: we use python 3.8 though the python scripts will probably work with 3.7. The following python packages are required. The list shows the versions we are using as of June, 2020.
  • cryptography: version 2.9.2
  • numpy: version 1.18.5
  • pandas: version 1.0.4
  • pexpect: version 4.8.0
  • Pillow: version 7.1.2
  • dash: version 1.12.0, used for the interactive plotting web app
  • uwsgi: version 2.0.18, used for the interactive plotting web app
• Fortran compiler: we currently use gfortran from gcc.gnu.org
• C compiler: we use the clang compiler included with Xcode

Running the pipeline

The pipeline is scheduled to run each day at 16:32 using launchctl. The pipeline can be started manually with the command

sudo launchctl start get_mrt

You can also define the VOY and fky environment variables as described above and run

${VOY}/bin/getmrt.py new --filter

from the ${VOY} directory.

getmrt.py overview

Run getmrt.py -h to get help on using the script. The script is run with the new and --filter option when started by launchctl

1. If the command line argument "new" is used, the getmrt.py script downloads new zipped mrt files from APL to ${VOY}/mrt/zip/ using rsync. If a specific mrt file date, syyddd, is used instead, rsync downloads the specified mrt file to the ${VOY}/mrt/zip/ directory if the file exists at APL. Login is handled by the pexpect python package. If no files are downloaded the script sends an email message and exits.

2. Files that pass a simple Cruise5A check are unzipped in the ${VOY}/mrt/zip directory. The compiled c program reform is used to change the record format of the unzipped mrt files to one that can be read by the gfortran compiled programs. The converted files are moved to the ${VOY}/mrt directory.

3. The names of the new files are used to determine the year, first day and last day arguments used to run the filter_run bash script. filter_run runs the compiled Fortran program filter2 to extract valid mrt records from the new mrt files and save them to yearly mrt files for each spacecraft in the filt directory.

4. getmrt runs the averages bash script to update the 1, 26, and 52 day and yearly flux database files and make the time-intensity plots. averages runs the voflx_avg bash script which runs the voflx_run bash script. voflx_run runs the compiled Fortran program voflx_cr5a to calculate the fluxes. voflx_avg then runs the vfmaster bash script. vfmaster runs the compiled Fortran program update_master to update the database files from the .ovf files created by voflx_cr5a. vfmaster then runs the autoplot bash script to generate the time-intensity plots using the plotsums.pro IDL script.

5. getmrt runs the webplot bash script to create the 26 and 52 day and yearly spectra plots. webplot runs the autoplot bash script which uses the plotsums.pro IDL script to generate the spectra plots.

6. getmrt runs the autoplot_h31.sh bash script to generate the special static 1 day time-intensity plots. It also updates the 1 pandas dataframe files used by the interactive plotting web app.