MetaReader¶

class MetaReader(settings: Optional[dict] = None)

This class, MetaReader, is the base class for all things related to reading raw nanopore timeseries datafiles. It handles mapping groups of files that belong in the same experiment, separating them by channel in the case of multichannel experimental operations, and time-ordering files within a channel when many data files are written as part of a single experiment. Subsequently, it provides a common API through which to interact with that data, effectively standardizing data reading operations regardless of the source. Given the number of different file formats commonly in use in the nanopore field, this plugin will likely always have the largest number of subclasses.

What you get by inheriting from MetaReader¶

Regardless of the details of how your data is actually stored, MetaReader will provide a common and intuitive API with which to interact with it, stitching together all the files in your dataset to work seamlessly together as a single dataset. Datasets are broken down by channel ID and time, allowing slicing into data that might be spread across multiple files as though it were a single contiguous memory structure. Data can be retrieved either on an ad-hoc basis, or as a continuous generator that allows you to iterate through on demand. Metadata like sampling rate, the length of data available in each channel, etc., can be retrieved through the API directly.

Attributes:: logger (logging.Logger): Logger instance for logging messages.

Public Methods¶

Abstract Methods¶

These methods must be implemented by subclasses.

abstractmethod MetaReader.close_resources(channel: int | None = None) → None¶

Purpose: Clean up any open file handles or memory.

This is called during app exit or plugin deletion to ensure proper cleanup of resources that could otherwise leak. If channel is not None, handle only that channel, else close all of them. If no such operation is needed, it suffices to pass. Note that readers that operate based on memmaps need not explicitly close those memmaps, as they will be handled by the garbage collector, but it does no harm to do so. Any open file handles should be closed explicitly if not closed at the end of read operations.

Parameters:: channel (Optional[int]) – channel ID

abstractmethod MetaReader.reset_channel(channel: int | None = None) → None¶

Perform any actions necessary to gracefully close resources before app exit. Purpose: Reset the state of a specific channel for a new operation or run.

This is called any time an operation on a channel needs to be cleaned up or reset for a new run. If channel is not None, handle only that channel, else close all of them. If reading through a channel does not create any persistent state changes in your plugin, you can simply pass this function.

Parameters:: channel (Optional[int]) – channel ID

Concrete Methods¶

MetaReader.continuous_read(start: float = 0, total_length: float = 0, channel: int = 0, chunk_length: float = 0) → Generator[ndarray[tuple[int, ...], dtype[float64]], None, None]¶

Read rescaled data in chunks and yield it.

Every chunk is in pA; the unscaled continuous_read_raw was removed in 2.0.0.

Parameters:

start (float) – Start time in the timeseries data, in seconds (default is 0).
total_length (float) – Length of data to read, in seconds (default is 0, meaning to the end of the data).
channel (int) – channel index to analyze.
chunk_length (float) – Length of the data chunks to process at a time, in seconds (default is 0, auto-determined).

Yield:

successive chunks of rescaled data.

Ytype:

npt.NDArray[np.float64]

MetaReader.force_serial_channel_operations() → bool¶

Purpose: Indicate whether operations on different channels must be serialized (not run in parallel).

By default this simply returns False, meaning that it is acceptable and thread-safe to run operations on different channels in different threads on this plugin. If such operation is not thread-safe, this function should be overridden to simply return True. Most readers are thread-safe since reading from a file on disk is usually so, and therefore no override is necessary.

Returns:: True if only one channel can run at a time, False otherwise
Return type:: bool

MetaReader.get_base_experiment_name() → str¶

Get the base name of the experiment being analyzed

Returns:: name of the experiment being analyzed
Return type:: str

MetaReader.get_base_file() → PathLike¶

Return the full path to the file used to initiate this reader

Returns:: path to the file used to initiate the reader
Return type:: os.PathLike

MetaReader.get_channel_length(channel: int) → int¶

Return the number of samples in a channel.

Parameters:: channel (int) – Channel number to get length for.
Returns:: Number of samples in the specified channel.
Return type:: int

MetaReader.get_channels() → List[int]¶

Return the number of channels in the reader

Returns:: Number of channels in the reader
Return type:: List[int]

MetaReader.get_empty_settings(globally_available_plugins: Dict[str, List[str]] | None = None, standalone: bool = False) → Dict[str, Dict[str, Any]]¶

Purpose: Provide a list of settings details to users to assist in instantiating an instance of your MetaReader subclass.

Get a dict populated with keys needed to initialize the filter if they are not set yet. This dict must have the following structure, but Min, Max, and Options can be skipped or explicitly set to None if they are not used. Type is required; Value may be omitted or set to None, both meaning there is no default and the user must supply one. All values provided must be consistent with Type.

settings = {'Parameter 1': {'Type': <int, float, str, bool>,
                                  'Value': <value> or None,
                                  'Options': [<option_1>, <option_2>, ... ] or None,
                                  'Min': <min_value> or None,
                                  'Max': <max_value> or None,
                                  'Units': <unit str> or None
                              },
                              ...
                              }

Several parameter keywords are reserved: these are

‘Input File’ ‘Output File’ ‘Folder’ and all MetaClass names

These must have Type str and will cause the GUI to generate appropriate widgets to allow selection of these elements when used.

This function must implement returning of a dictionary of settings required to initialize the filter, in the specified format. Values in this dictionary can be accessed downstream through the self.settings class variable. This structure is a nested dictionary that supplies both values and a variety of information about those values, used by poriscope to perform sanity and consistency checking at instantiation.

While this function is technically not abstract in MetaReader, which already has an implementation of this function that ensures that settings will have the required MetaReader key available to users, in most cases you will need to override it to add any other settings required by your subclass. The implementation in MetaReader provides a single key, Input File, without specifing file options. If you need additional settings, or if you want to specify the file types that will show up in related file dialogs and be accepted as inputs (recommended) you would override this to specidy options, but you MUST call settings = super().get_empty_settings(globally_available_plugins, standalone) first, which will ensure the existence of the “Input File” key. For example:

settings = super().get_empty_settings(globally_available_plugins, standalone)
settings["Input File"]["Options"] = ["ABF2 Files (*.abf)"]
settings["Your Key"] = {"Type": float,
                        "Value": None,
                        "Min": 0.0,
                        "Units": "pA"
                        }
return settings

which will ensure that your have key specified above, as well as an additional key, Input File, as required by readers. You can learn more about formatting input file option strings in the PySide6 module documentation. In the case of multiple file types, supply the relevant strings as a comma-separated list in the “Options” key; poriscope will handle formatting it for PySide6.

Parameters:

globally_available_plugins (Optional[Dict[str, List[str]]]) – a dict containing all data plugins that exist to date, keyes by metaclass
standalone (bool) – True if this is outside the context of a GUI, False otherwise, Default False.

Returns:

the dict that must be filled in to initialize the filter

Return type:

Dict[str, Dict[str, Any]]

MetaReader.get_samplerate() → float¶

Return the sampling rate for the dataset.

Returns:: Sampling rate for the dataset.
Return type:: float

MetaReader.load_data(start: float, length: float, channel: int = 0) → ndarray[tuple[int, ...], dtype[float64]]¶

Return data starting at start and of length seconds, rescaled to pA.

Every read returns pA. Until 2.0.0 a raw_data flag, and then a separate load_raw_data, returned the unscaled samples instead; that path is gone, because a float64 reading loses nothing that matters and nothing used it.

Validation happens in _slice_request(), which raises ValueError for an out-of-bounds or uncoercible request and IndexError when the channel has no data map or config. Both propagate out of this method unchanged.

Parameters:

start (float) – Start time of the data to load, in seconds.
length (float) – Length of data to load, in seconds.
channel (int) – Channel number from which to load data.

Returns:

Converted and rescaled data.

Return type:

npt.NDArray[np.float64]

MetaReader.report_channel_status(channel: int | None = None, init: bool = False) → str¶

Return a string detailing any pertinent information about the status of analysis conducted on a given channel

Parameters:

channel (Optional[int]) – channel ID
init (bool) – is the function being called as part of plugin initialization? Default False

Returns:

the status of the channel as a string

Return type:

str

Private Methods¶

Abstract Methods¶

These methods must be implemented by subclasses.

abstractmethod MetaReader._convert_data(data: ndarray[tuple[int, ...], dtype[int16]], config: dict) → ndarray[tuple[int, ...], dtype[float64]]¶

Purpose: Convert raw data from disk into rescaled floating-point current.

Given a numpy array of raw data taken from one of the memmap instances you built in _map_data(), along with the config dict that belongs to the same file, return it as float64 in pA. Derive whatever scale, offset and bitmask the format needs from config and hand them to _scale_data(), which applies them in the right order:

def _convert_data(self, data, config):
    scale = 1e12 * 2 * config["v_ref"] / (2**16 * config["tia_gain"])
    offset = -config["i_offset"] * 1e12
    return self._scale_data(
        data,
        scale=scale,
        offset=offset,
        dtype=np.float64,
        copy=False,
    )

This is the only conversion a reader provides. Until 2.0.0 readers also implemented _convert_raw_data (the same bytes unscaled, with the factors alongside) and _set_raw_dtype; both are gone with the raw-data path. A reader that needs its on-disk dtype builds it privately, for use in _map_data().

Parameters:

data (npt.NDArray[np.int16]) – Data to convert.
config (dict) – Configuration dictionary for data conversion.

Returns:

The data, rescaled to pA.

Return type:

npt.NDArray[np.float64]

abstractmethod MetaReader._get_configs(datafiles: List[str]) → List[dict]¶

Purpose: Extract configuration metadata from dataset files.

Given a list of filenames corresponding to the data files, construct a list of dictionaries containing any required configurations for use downstream. Your config dictionaries must have at a minimum the key ‘samplerate’ in them, and the list of configs must correspond one-to-one to the provided list of data files. All files in a dataset must have the same samplerate. Your reader will use these configs to map the data on disk, so you could include information like endianness, raw data type, details of any columns within the data, etc. Aside from the required samplerate key, this can be anything.

Parameters:: datafiles (List[str]) – List of data files for which to load configurations.
Returns:: List of configuration dictionaries.
Return type:: List[dict]

abstractmethod MetaReader._get_file_channel_stamps(file_names: List[str], configs: List[dict]) → List[int]¶

Purpose: Extract channel identifiers for grouping files by channel.

Given a list of all the files in the experiment and the list of config dictionaries you defined above, extract a corresponding list of channel identifiers as integers. These channel indices will be used to group the mapped data by channel. The list must have the same length as both input lists and must be a list of integers.

Parameters:

file_names (List[str]) – List of file names to get channel stamps for.
configs (List[dict]) – List of configuration dictionaries corresponding to data files.

Returns:

List of serialization keys for channels

Return type:

List[int]

abstractmethod MetaReader._get_file_pattern(file_name: str) → str¶

Purpose: Extract a glob pattern from an input filename to match all dataset files.

When you instantiate a reader plugin, you provide a single filename as input. However, in some cases, a dataset might comprise many files. This function requires you to extract a pattern from the given filename that can be used by glob to match all files belonging to your dataset.

If your dataset consists of only a single file, you can simply return the original filename.

Example:

Consider a scenario where your dataset files follow a pattern with channel numbers and serial numbers, such as:

experiment_1_channel_01_001.log
experiment_1_channel_01_002.log
experiment_1_channel_02_001.log
experiment_1_channel_02_002.log

In this case, you could return a glob pattern like:

experiment_1_channel_??_???.log

This pattern assumes the channel stamp will always be two digits and the serial number always three. If the lengths of these varying parts are uncertain, a more general pattern using wildcards would be:

experiment_1_channel_*_*.log

Poriscope will use this file pattern to search the folder of the input file for other files that match the pattern. It will not search outside of that folder.

Anchor the pattern on what follows the base name rather than ending it in a bare *: experiment_1*.log also matches experiment_10_channel_01_001.log, a different recording, and its files are then read as part of this one. Pass the literal part through glob.escape(), so that a name containing [ or ] matches itself: glob.escape("experiment_1") + "_channel_*_*.log".

For more information on glob patterns, refer to the glob module documentation.

Parameters:: file_name (str) – File name to get the base pattern for.
Returns:: Base pattern for matching other files.
Return type:: str

abstractmethod MetaReader._get_file_time_stamps(file_names: List[str], configs: List[dict]) → Any¶

Purpose: Extract time stamps for sorting files chronologically within a channel.

Given a list of all the files in the experiment and the list of config dictionaries you defined above, extract a corresponding list of timestamps. These timestamps will be used to time-order the mapped data within each channel. The list must have the same length as both input lists and must be of a type that can be sorted into the desired time-ordering using the builtin sort() method.

Parameters:

file_names (List[str]) – List of file names to get time stamps for.
configs (List[dict]) – List of configuration dictionaries corresponding to data files.

Returns:

List of serialization keys for timestamps in almost any format (e.g. str, int, float, datetime.datetime, datetime.date, or np.datetime64); the concrete element type is defined by the subclass.

Return type:

Any

abstractmethod MetaReader._init() → None¶

Purpose: Perform generic class construction operations.

All data plugins have this function and must provide an implementation. This is called immediately at the start of class creation and is used to do whatever is required to set up your reader. Note that no app settings are available when this is called, so this function should be used only for generic class construction operations. Most readers simply pass this function.

abstractmethod MetaReader._map_data(datafiles: List[str], configs: List[dict]) → List[ndarray]¶

Purpose: Map the provided data files into an accessible format, preferably memory-mapped views.

Using all the information provided in the implementations so far, in this function, you are asked to map the list of files provided in datafiles, according to information given in configs. You can assume that the lists are of equal length and that the config file at a given index corresponds to the data file at the same index. You must return a list of views into those files. We strongly encourage the use of memmap where possible, in which case you may return a list of such memmaps with length equal to the input list of filenames.

Warning

This function expects that the elements of the returned list can be indexed and sliced into like NumPy arrays, hence the suggestion to use memmaps, which avoid the need to actually load raw data into RAM before it is needed. In cases where memmap is not an option, you must still return NumPy array for each file, which may involve significant memory consumption. If this is impractical, it is possible to override this function to return, for example, a list of file handles instead, with the caveat that this will in turn require that you completely override load_data() as well to properly handle your file access method manually.

Parameters:

datafiles (List[str]) – List of data files to map.
configs (List[dict]) – List of configuration dictionaries corresponding to data files.

Returns:

List of memmaps or numpy arrays mapped from data files.

Return type:

List[np.ndarray]

abstractmethod MetaReader._set_file_extension() → str¶

Purpose: Set the file extension for the file type this reader plugin handles.

This is a simple function that allows you to set the file extension (including the leading dot) of the file type that this reader plugin will read. It is used by downstream functions while mapping your data to assist in identifying files. It should be a single line:

return ".ext"

If you need to refer to this value again, you can access it via the class variable self.file_extension.

Returns:: the file extension
Return type:: str

abstractmethod MetaReader._validate_file_type(filename: PathLike) → None¶

Check that the file(s) being opened are of the correct type, and raise IOError if not

Parameters:: filename (os.PathLike) – the path to one of the files to be opened
Raises:: IOError – If the wrong type is file is fed to the plugin

abstractmethod MetaReader._validate_settings(settings: dict) → None¶

Validate that the settings dict contains the correct information for use by the subclass.

Parameters:: settings (dict) – Parameters required to configure this reader.
Raises:: ValueError – If the settings dict does not contain the correct information.