mDOS parser

Overview

The mDOS parser is a tool to read and convert the survey data taken with one of our MS-xxx soil/sediment mapping sensors. The use of the tool is very straightforward:

1. Select the data files you've downloaded from the sensor;
2. Select one or more output formats;
3. Select an optional template for parsing;
4. Click the Parse button.

Download

Download the installer here: mDOS_parser_setup.exe

Parsing data

Getting ready to parse

The steps you need to take to parse data are as follows:

1. Select a folder containing the recorded JSON files. Note that all selected JSON files must be recorded using the same system, as it is not possible to merge data recorded with different systems into a single project;
2. Choose which file formats to export to:
   1. Gamman GSF project file → this will produce a GSF (Gamman Survey File) that can be used right away in Gamman;
   2. CSV file → this will create a comma-separated file containing all fields that are selected in the template. Note that you can select a delimiter different from the comma in the preferences section;
   3. MGP file → this will build a Medusa Gamman Project file, which is a JSON file that contains the selected data in a coherent format that can be used for post-processing.
3. Select the template you want to use. The last used template is stored when you close the program and will be selected when you run the software again. More on templates can be found in the Data files and templates section further on this page;
4. Click the Parse button.
5. Choose a filename to write to. 

Reading and sorting the data

The first step of the parser is to read all JSON files and create a list in memory of all recorded measurement data from these files. The list will then be sorted on time. This can be a lengthy process, depending on the amount of data you have selected for processing. Check the section with parsing messages, as it will notify any errors that it has encountered, such as invalid or corrupted lines in the input files. 

Synchronisation of data

The second task of the parser is to synchronise all "auxiliary" data to the gamma-ray spectra measured by the spectrometer. This synchronisation is needed since all data is logged at its own pace. For instance, it can be your GPS is set to record positions every 0,1 second. The spectra are normally taken once per second, so the GPS positions are not necessarily taken at exactly the same moment in time as the spectrometer. The synchroniser basically runs through the list of spectra, and for each spectrum it calculates the auxiliary data for the timestamp of the spectrum. This is done through linear interpolation over the values closest in time before and after the timestamp of the spectrum. In case no auxiliary data has been recorded within a period of 30 seconds before and after the spectrum, this data is omitted from the output.

Outputting data

After the synchronisation has finished, the data is exported and written to a GSF, CSV and/or MGP file.

Data files and templates

The data files produced by the Medusa Soil Mapping Systems (SMS) are stored in JSON format. JSON is a structured and human-readable data formatting system that is used widely in modern data acquisition systems. The JSON files consist of a header part that contains info on the sensor and on the data it is capable of gathering. The rest of the file contains the measurement data resulting from your survey. Each line is an individual JSON record containing the data taken at a certain moment in time. This data can be from the embedded GPS, from the embedded pressure, temp and humidity sensors, or from external instruments. And the file contains the output of the gamma-ray spectrometer which is at the heart of your SMS. 

The output of a Medusa system can be pretty verbose and may contain all kinds of info you do not need. You can use a template to specify what data you want to see in the output, and what data can be omitted. It is possible to automatically generate a default template while importing, which will include all unprocessed data, but omits processed data such as the stabilized spectrum and nuclide concentrations. 

Managing templates

To manage the templates, click the Manage templates button and the form as shown below will be visible. The list on the left side of the form contains all existing templates. By selecting a template, the contents will be shown on the main section of the display. In this case, the template will export data from the GPS, the PTH and the Spectrometer sensors. By clicking on a sensor, the quantities to be exported will be shown. For the "PTH" sensor, the pressure, temperature and humidity will be exported. Clicking on a quantity will shown some additional information on the quantity.

A new template can be made, based on a recorded JSON file. To do, this click the New button and select a JSON file. After selecting a JSON file, it is read and a dialog as shown below will pop up.

The dialog contains a list of the sensors and all quantities in the JSON file. By click on a sensor or a quanity, additional information will be shown. While parsing data based on this template, only the checked quanties will be added to the output. It is important to select a quantity to synchronize on, as this quantity is used as an identifier to synchronize all other quantities to.

It is possible to generate a template that matches sensors with similar configuration by checking the Match all serial numbers option. In the screenshot above, this will create a template that also parses data not only for PTH_0009m but for all sensors ranging from  PTH_0001 to PTH_9999. If the option is left unchecked, and the template is used to parse a JSON file that contains data from PTH_0010, this data will not be added to the output, as it does not match the required sensor name.

An existing template can be edited similarly to creating a new template. To edit a template, select it in the template manager and click the Edit button. Exporting and importing templates can be done using the Import and Export buttons in the template manager.

SPECTRO or STABSPECTRO?

The latest versions of the medusa Detector Operating System (mDOS) store spectra both in a "raw" format (as 512 channels) and in an energy-stabilised format. In the latter, each spectrum is shifted in such a way that all peaks arrive at the proper gamma-ray energy. This process is called "energy-stabilisation" and used to be a procedure that could only be done in post-processing (using Gamman). Now, you can choose whether to use the "raw" spectra, or the energy-stabilised ones. You can export the the stabilised spectra by checking STABSPECTRO_xxx in the template manager, and the raw data by checking SPECTRO_xxx.

{}