e15190.runlog.data_cleansing.ElogCleanser

class e15190.runlog.data_cleansing.ElogCleanser(elog_path=None, read_elog=True)

Functions

__init__([elog_path, read_elog])

To cleanse the ELOG downloaded from the web.

cleanse([cleanse_runs, cleanse_events])

Cleanse the elog.

filtered_runs()

Updates and returns filtered experimental runs.

read(**kwargs)

save_cleansed_elog([output_dir, df_runs, ...])

Save experimental runs and events after data cleansing.

save_filtered_runs(file_extension[, ...])

Save filtered experimental runs to a file.

Properties

CLEANSED_DIR

OUTPUT_DIR

events_final

Dataframe of events after filtering

__init__(elog_path=None, read_elog=True)

To cleanse the ELOG downloaded from the web.

The ELOG is hosted at http://neutronstar.physics.wmich.edu/runlog/index.php?op=list It is basically a large table that contains the following columns:

  • RUN

  • Begin time

  • End time

  • Elapse

  • DAQ

  • Type

  • Target

  • Beam

  • Shadow bars

  • Trigger rate

  • Comments

Each row is either an experimental run or an event.

The purpose of this class is to ensure all columns have been converted into some suitable data types. Some simple removal of entries (rows) is also done here.

Parameters
  • elog_path (str, default None) – The path to the local ELOG html file. If None, the default path is used, i.e. $DATABASE_DIR/runlog/downloads/elog.html.

  • read_elog (bool, default True) – Read in the HTML file.

events_final

Dataframe of events after filtering

Type

To do

cleanse(cleanse_runs=True, cleanse_events=True)

Cleanse the elog.

This function updates the self.runs and self.events dataframes. Most entries are being preserved, i.e. minimal filtering is done except for those that are labeled as “junk” or simply corrupted.

save_cleansed_elog(output_dir=None, df_runs=None, df_events=None, elog_hdf_name='elog', elog_runs_csv_name='elog_runs', elog_events_csv_name='elog_events', verbose=True)

Save experimental runs and events after data cleansing.

Both self.runs and self.events are saved into one single HDF5 file. Then they are saved into two separate CSV files.

HDF5 file has the advantage of preserving the data types, while CSV files allow users to inspect the data with simple text editor or spreadsheet software.

Parameters
  • output_dir (str or pathlib.Path, default None) – The directory to save the files. If None, the default path is used, i.e. $PROJECT_DIR/database/runlog/cleansed.

  • df_runs (pd.DataFrame, default None) – Experimental runs to save. If None, the default is to use self.runs.

  • df_events (pd.DataFrame, default None) – Events to save. If None, the default is to use self.events.

  • elog_hdf_name (str, default 'elog') – The name of the HDF file without the extension; the extension .h5 will be appended automatically.

  • elog_runs_csv_name (str, default 'elog_runs') – The name of the CSV file for experimental runs without the extension; the extension .csv will be appended automatically.

  • elog_events_csv_name (str, default 'elog_events') – The name of the CSV file for events without the extension; the extension .csv will be appended automatically.

  • verbose (bool, default True) – Whether to print the progress.

_cleanse_runs()

Cleanse the entries (rows) of experimental runs.

This function updates the self.runs dataframe and go through the following steps:

  • Remove entries whose “Type” are labeled as “junk”.

  • Convert all runs into integers.

  • Convert all times into datetimes, this includes “Begin time” and “End time”.

  • Convert elapse time into timedelta, this includes “Elapse”.

  • Standardize the “Target” into one of “Ni58”, “Ni64”, “Sn112”, “Sn124” or “CH2”.

  • Turn all “nan” in “Type”, “Target”, “Beam” and “Shadow bars” into empty strings.

  • Convert “Trigger rate” into float.

  • Convert “Comments” into “none” if it is “nan”.

  • Use singular form for all column names, e.g. “Shadow bars” -> “Shadow bar”.

  • Convert all column names to lowercase, and replace spaces with underscores.

_cleanse_events()

Cleanse the entries (rows) of events.

This function updates the self.events dataframe which has columns “run”, “time”, “entry” and “comment”.

filtered_runs()

Updates and returns filtered experimental runs.

This function filters the experimental runs and keeps only the meaningful entries. The filters are:

  • run in [2000, 3000) or [4000, 5000)

  • elapse time > 5 minutes

  • daq == “Merged”

  • type == “data”

  • target is one of “Ni58”, “Ni64”, “Sn112” or “Sn124”

  • beam is one of “Ca40 56 MeV/u”, “Ca40 140 MeV/u”, “Ca48 56 MeV/u” or “Ca48 140 MeV/u”

  • trigger rate > 1000.0 / sec

Returns

runs_final – Filtered experimental runs.

Return type

pd.DataFrame

save_filtered_runs(file_extension, output_path=None, df=None, verbose=True)

Save filtered experimental runs to a file.

HDF5 file has the advantage of preserving the data types, while CSV files allow users to inspect the data with simple text editor or spreadsheet software.

Parameters
  • file_extension ('csv' or 'h5') – File extension of the output file. This will be overwritten if output_path is specified explicitly by user.

  • output_path (pathlib.Path, default None) – Path to the output file. If None, the default path is $PROJECT_DIR/database/runlog/elog_runs_filtered.${EXT}, where ${EXT} is the file extension specified by file_extension.

  • df (pd.DataFrame, default None) – Experimental runs to save. If None, the default is to use self.runs_final.

  • verbose (bool, default True) – Whether to print the message.

Raises

ValueError – If file_extension is not one of ‘csv’ or ‘h5’.

static _save_as_csv(df, filepath)

Save a dataframe as a csv file.

Parameters
  • df (pd.DataFrame) – Dataframe to save.

  • filepath (str or pathlib.Path) – Path to the output file.

static _save_as_hdf(df, filepath)

Save a dataframe as an hdf file.

Parameters
  • df (dict of pd.DataFrame) – A dictionary of dataframes to save.

  • filepath (str or pathlib.Path) – Path to the output file.