The daily processing pipeline at UMD downloads new mrt files from APL and runs python, bash and IDL scripts and compiled Fortran programs to update various plots and ascii database files available on the UMD Voyager web page. This document describes the pipeline as of May, 2020.
The pipeline was originally run on a DEC VAXstation(??) using compiled DEC Fortran code, IDL scripts and DCL batch files. When the VAXstation was decommissioned the pipeline and web pages were moved to a powerpc based Mac. Larry Bleau and Matt Hill ported the DEC Fortran code to compile with the g95 compiler from g95.org on a powerpc Mac, and ported the DCL scripts to bash scripts. When the pipeline was moved to an intel Mac mini the Fortran code was updated to work on little endian processors and to compile with the GNU gfortran compiler. The get_mrt bash script was replaced with the getmrt.py python script in order to add new features to the pipeline such as interactive plotting of fluxes and rates.
The pipeline is scheduled to run each day at 16:32 using launchctl. The pipeline can be started manually with the command
sudo launchctl start get_mrt
You can also define the VOY and fky environment variables as described above and run
${VOY}/bin/getmrt.py new --filter
from the ${VOY} directory.
Run getmrt.py -h to get help on using the script. The script is run with the new and --filter option when started by launchctl
If the command line argument "new" is used, the getmrt.py script downloads new zipped mrt files from APL to ${VOY}/mrt/zip/ using rsync. If a specific mrt file date, syyddd, is used instead, rsync downloads the specified mrt file to the ${VOY}/mrt/zip/ directory if the file exists at APL. Login is handled by the pexpect python package. If no files are downloaded the script sends an email message and exits.
Files that pass a simple Cruise5A check are unzipped in the ${VOY}/mrt/zip directory. The compiled c program reform is used to change the record format of the unzipped mrt files to one that can be read by the gfortran compiled programs. The converted files are moved to the ${VOY}/mrt directory.
The names of the new files are used to determine the year, first day and last day arguments used to run the filter_run bash script. filter_run runs the compiled Fortran program filter2 to extract valid mrt records from the new mrt files and save them to yearly mrt files for each spacecraft in the filt directory.
getmrt runs the averages bash script to update the 1, 26, and 52 day and yearly flux database files and make the time-intensity plots. averages runs the voflx_avg bash script which runs the voflx_run bash script. voflx_run runs the compiled Fortran program voflx_cr5a to calculate the fluxes. voflx_avg then runs the vfmaster bash script. vfmaster runs the compiled Fortran program update_master to update the database files from the .ovf files created by voflx_cr5a. vfmaster then runs the autoplot bash script to generate the time-intensity plots using the plotsums.pro IDL script.
getmrt runs the webplot bash script to create the 26 and 52 day and yearly spectra plots. webplot runs the autoplot bash script which uses the plotsums.pro IDL script to generate the spectra plots.
getmrt runs the autoplot_h31.sh bash script to generate the special static 1 day time-intensity plots. It also updates the 1 pandas dataframe files used by the interactive plotting web app.