e15190.runlog.data_cleansing.ElogCleanser¶
- class e15190.runlog.data_cleansing.ElogCleanser(elog_path=None, read_elog=True)¶
Functions
__init__([elog_path, read_elog])To cleanse the ELOG downloaded from the web.
cleanse([cleanse_runs, cleanse_events])Cleanse the elog.
Updates and returns filtered experimental runs.
read(**kwargs)save_cleansed_elog([output_dir, df_runs, ...])Save experimental runs and events after data cleansing.
save_filtered_runs(file_extension[, ...])Save filtered experimental runs to a file.
Properties
CLEANSED_DIROUTPUT_DIRDataframe of events after filtering
- __init__(elog_path=None, read_elog=True)¶
To cleanse the ELOG downloaded from the web.
The ELOG is hosted at http://neutronstar.physics.wmich.edu/runlog/index.php?op=list It is basically a large table that contains the following columns:
RUN
Begin time
End time
Elapse
DAQ
Type
Target
Beam
Shadow bars
Trigger rate
Comments
Each row is either an experimental run or an event.
The purpose of this class is to ensure all columns have been converted into some suitable data types. Some simple removal of entries (rows) is also done here.
- Parameters
elog_path (str, default None) – The path to the local ELOG html file. If
None, the default path is used, i.e.$DATABASE_DIR/runlog/downloads/elog.html.read_elog (bool, default True) – Read in the HTML file.
- events_final¶
Dataframe of events after filtering
- Type
To do
- cleanse(cleanse_runs=True, cleanse_events=True)¶
Cleanse the elog.
This function updates the
self.runsandself.eventsdataframes. Most entries are being preserved, i.e. minimal filtering is done except for those that are labeled as “junk” or simply corrupted.
- save_cleansed_elog(output_dir=None, df_runs=None, df_events=None, elog_hdf_name='elog', elog_runs_csv_name='elog_runs', elog_events_csv_name='elog_events', verbose=True)¶
Save experimental runs and events after data cleansing.
Both
self.runsandself.eventsare saved into one single HDF5 file. Then they are saved into two separate CSV files.HDF5 file has the advantage of preserving the data types, while CSV files allow users to inspect the data with simple text editor or spreadsheet software.
- Parameters
output_dir (str or pathlib.Path, default None) – The directory to save the files. If
None, the default path is used, i.e.$PROJECT_DIR/database/runlog/cleansed.df_runs (pd.DataFrame, default None) – Experimental runs to save. If None, the default is to use
self.runs.df_events (pd.DataFrame, default None) – Events to save. If None, the default is to use
self.events.elog_hdf_name (str, default 'elog') – The name of the HDF file without the extension; the extension
.h5will be appended automatically.elog_runs_csv_name (str, default 'elog_runs') – The name of the CSV file for experimental runs without the extension; the extension
.csvwill be appended automatically.elog_events_csv_name (str, default 'elog_events') – The name of the CSV file for events without the extension; the extension
.csvwill be appended automatically.verbose (bool, default True) – Whether to print the progress.
- _cleanse_runs()¶
Cleanse the entries (rows) of experimental runs.
This function updates the
self.runsdataframe and go through the following steps:Remove entries whose “Type” are labeled as “junk”.
Convert all runs into integers.
Convert all times into datetimes, this includes “Begin time” and “End time”.
Convert elapse time into timedelta, this includes “Elapse”.
Standardize the “Target” into one of “Ni58”, “Ni64”, “Sn112”, “Sn124” or “CH2”.
Turn all “nan” in “Type”, “Target”, “Beam” and “Shadow bars” into empty strings.
Convert “Trigger rate” into float.
Convert “Comments” into “none” if it is “nan”.
Use singular form for all column names, e.g. “Shadow bars” -> “Shadow bar”.
Convert all column names to lowercase, and replace spaces with underscores.
- _cleanse_events()¶
Cleanse the entries (rows) of events.
This function updates the
self.eventsdataframe which has columns “run”, “time”, “entry” and “comment”.
- filtered_runs()¶
Updates and returns filtered experimental runs.
This function filters the experimental runs and keeps only the meaningful entries. The filters are:
run in [2000, 3000) or [4000, 5000)
elapse time > 5 minutes
daq == “Merged”
type == “data”
target is one of “Ni58”, “Ni64”, “Sn112” or “Sn124”
beam is one of “Ca40 56 MeV/u”, “Ca40 140 MeV/u”, “Ca48 56 MeV/u” or “Ca48 140 MeV/u”
trigger rate > 1000.0 / sec
- Returns
runs_final – Filtered experimental runs.
- Return type
pd.DataFrame
- save_filtered_runs(file_extension, output_path=None, df=None, verbose=True)¶
Save filtered experimental runs to a file.
HDF5 file has the advantage of preserving the data types, while CSV files allow users to inspect the data with simple text editor or spreadsheet software.
- Parameters
file_extension ('csv' or 'h5') – File extension of the output file. This will be overwritten if
output_pathis specified explicitly by user.output_path (pathlib.Path, default None) – Path to the output file. If None, the default path is
$PROJECT_DIR/database/runlog/elog_runs_filtered.${EXT}, where${EXT}is the file extension specified byfile_extension.df (pd.DataFrame, default None) – Experimental runs to save. If None, the default is to use
self.runs_final.verbose (bool, default True) – Whether to print the message.
- Raises
ValueError – If
file_extensionis not one of ‘csv’ or ‘h5’.
- static _save_as_csv(df, filepath)¶
Save a dataframe as a csv file.
- Parameters
df (pd.DataFrame) – Dataframe to save.
filepath (str or pathlib.Path) – Path to the output file.
- static _save_as_hdf(df, filepath)¶
Save a dataframe as an hdf file.
- Parameters
df (dict of pd.DataFrame) – A dictionary of dataframes to save.
filepath (str or pathlib.Path) – Path to the output file.