MetaWriter¶

class MetaWriter(settings: Optional[dict] = None)

What you get by inheriting from MetaWriter¶

MetaWriter is the base class for writing the data corresponding to events found by a MetaEventFinder subclass instance events within your nanopore data and represents the first analysis and transformation step. MetaWriter depends on and is linked at instantiation to a MetaEventFinder subclass instance that serves as its source of nanopore data, meaning that creating and using one of these plugins requires that you first instantiate an eventfinder.

Poriscope ships with SQLiteEventWriter, a subclass of MetaWriter already that writes data to a sqlite3 format. While additional subclasses can write to almost any format you desire, we strongly encourage standardization around this format. Think twice before creating additional subclasses of this base class. It is not sufficient to write just a MetaWriter subclass. In addition to this base class, you will also need a paired MetaEventLoader subclass to read back and use the data you write to any other format for downstream analysis.

Warning

We strongly encourage standardization on the :ref:SQLiteDBWriter subclass, so please think carefully before creating other formats.

Public Methods¶

Abstract Methods¶

These methods must be implemented by subclasses.

abstractmethod MetaWriter.close_resources(channel: int | None = None) → None¶

Purpose: Clean up any open file handles or memory.

This is called during app exit or plugin deletion, as well as at the end of any batch write operation, to ensure proper cleanup of resources that could otherwise leak. Do this for all channels if no channel is specified, otherwise limit your closure to the specified channel. Your files should be closed here, if they are not in your writing step. If no such operation is needed, it suffices to pass. In the case of writers, this method is also called with a specific channel identifier at the end of any batch write operation (a call to commit_events()), and so should be used to ensure atomic write operations if possible.

Parameters:: channel (Optional[int]) – channel ID

abstractmethod MetaWriter.reset_channel(channel: int | None = None) → None¶

Purpose: Reset the state of a specific channel for a new operation or run.

This is called any time an operation on a channel needs to be cleaned up or reset for a new run. If channel is not None, handle only that channel, else close all of them. Most writers will create permanent state changes in the form of data written to the output file, that should be deleted or otherwise set up for subsequent overwrite when this function is called.

Parameters:: channel (Optional[int]) – channel ID

Concrete Methods¶

MetaWriter.commit_events(channel: int, overwrite: bool = False) → Generator[float, bool | None, None]¶

Create a generator that will loop through events in self.eventfinder in channel and call self._write_data() to commit it to file

If the output already holds channel (see get_committed_experiment_name()), the commit is refused with ValueError unless overwrite is True, in which case the channel is reset first. Writing into it anyway would mix two runs’ events in one channel. The Raw Data tab asks the user before passing overwrite.

Parameters:

channel (int) – the index of the channel to commit
overwrite (bool) – replace the channel’s events if the output already holds it

Yield:

the progress of the interator, normalized to [0,1]

Ytype:

float

MetaWriter.force_serial_channel_operations() → bool¶

Purpose: Indicate whether operations on different channels must be serialized (not run in parallel).

By default, writer plugins are assumed to not be threadsafe and will run in serial mode when called from the poriscope GUI. If you want to change this, you must also ensure that the parent eventfinder object is threadsafe for pulling data from it. You can play it safe by calling self.eventfinder.force_serial_channel_operations(), but it is possible that an eventfinder is not threadsafe for eventfinding but may be for pulling the events found for writing. How this is enforced: returning True means this plugin instance’s channel operations must not overlap each other; different instances still run concurrently. The guard is taken by the plugin itself, inside its own generator, via serialize_channels() and serialize_channel_operations(), and the lock is held for the whole run of that generator. It used to be enforced by the analysis tab’s model using a lock scoped to the model rather than to the plugin, which meant two tabs driving the same plugin took different locks and did not serialize at all.

Returns:: True if only one channel can run at a time, False otherwise
Return type:: bool

MetaWriter.get_committed_experiment_name(channel: int) → str | None¶

Name the experiment the output already holds channel under, if it does.

Asked before a commit, so it must not create the output file. The base returns None, meaning “this writer’s output never already holds a channel”; override it in a writer that can write into an existing file, so commit_events() can refuse to mix two runs in one channel.

Parameters:: channel (int) – the channel to look up
Returns:: the stored experiment name, or None if the output does not hold the channel
Return type:: Optional[str]

MetaWriter.get_empty_settings(globally_available_plugins: Dict[str, List[str]] | None = None, standalone: bool = False) → Dict[str, Dict[str, Any]]¶

Purpose: Provide a list of settings details to users to assist in instantiating an instance of your MetaWriter subclass.

Get a dict populated with keys needed to initialize the filter if they are not set yet. This dict must have the following structure, but Min, Max, and Options can be skipped or explicitly set to None if they are not used. Type is required; Value may be omitted or set to None, both meaning there is no default and the user must supply one. All values provided must be consistent with Type.

settings = {'Parameter 1': {'Type': <int, float, str, bool>,
                                 'Value': <value> or None,
                                 'Options': [<option_1>, <option_2>, ... ] or None,
                                 'Min': <min_value> or None,
                                 'Max': <max_value> or None
                                },
                ...
                }

This function must implement returning of a dictionary of settings required to initialize the writer, in the specified format. Values in this dictionary can be accessed downstream through the self.settings class variable. This structure is a nested dictionary that supplies both values and a variety of information about those values, used by poriscope to perform sanity and consistency checking at instantiation.

While this function is technically not abstract in MetaWriter, which already has an implementation of this function that ensures that settings will have the required MetaEventFinder key and Output File key available to users, in most cases you will need to override it to add any other settings required by your subclass. If you need additional settings, which you almost certainly do, you MUST call super().get_empty_settings(globally_available_plugins, standalone) before any additional code that you add. For example, your implementation could look like this:

settings = super().get_empty_settings(globally_available_plugins, standalone)
settings["Output File"]["Options"] = [
                        "SQLite3 Files (*.sqlite3)",
                        "Database Files (*.db)",
                        "SQLite Files (*.sqlite)",
                        ]
settings["Experiment Name"] = {"Type": str}
settings["Voltage"] = {"Type": float, "Units": "mV"}
settings["Membrane Thickness"] = {"Type": float, "Units": "nm", "Min": 0}
settings["Conductivity"] = {"Type": float, "Units": "S/m", "Min": 0}
return settings

which will ensure that your have the 4 keys specified above, as well as two additional keys, MetaReader and Output File. By default, it will accept any file type as output, hence the specification of the Options key for the relevant plugin in the example above.

Parameters:

globally_available_plugins (Optional[Dict[str, List[str]]]) – a dict containing all data plugins that exist to date, keyed by metaclass. Must include “MetaReader” as a key, with explicitly set Type MetaReader.
standalone (bool) – False if this is called as part of a GUI, True otherwise. Default False

Raises:

KeyError – if no MetaEventFinder has been instantiated and standalone is False

Returns:

the dict that must be filled in to initialize the filter

Return type:

Dict[str, Dict[str, Any]]

MetaWriter.get_output_file_name() → Path¶: get the name of the output file

MetaWriter.report_channel_status(channel: int | None = None, init: bool = False) → str¶

Return a string detailing any pertinent information about the status of analysis conducted on a given channel

Parameters:

channel (Optional[int]) – channel ID
init (bool) – is the function being called as part of plugin initialization? Default False

Returns:

the status of the channel as a string

Return type:

str

Private Methods¶

Abstract Methods¶

These methods must be implemented by subclasses.

abstractmethod MetaWriter._finalize_initialization() → None¶

Purpose: Perform generic class construction operations after settings are applied. This function is called at the end of the apply_settings() function to perform additional initialization specific to the algorithm being implemented.

Perform any initialization tasks required after settings are applied. You can access the values in the settings dict provided as needed in the class variable self.settings[key]['Value'] where key corresponds to the keys in the provided settings dict (as provided to apply_settings() or to the constructor). You can freely make class variables here and you can assume (if using the poriscope app) that this will only be called from a single thread. .

Should Raise if initialization fails.

abstractmethod MetaWriter._init() → None¶

Purpose: Perform generic class construction operations.

All data plugins have this function and must provide an implementation. This is called immediately at the start of class creation and is used to do whatever is required to set up your reader. Note that no app settings are available when this is called, so this function should be used only for generic class construction operations. Most readers simply pass this function.

abstractmethod MetaWriter._initialize_database(channel: int) → None¶

Purpose: Initialize a database for subsequent write operations.

This function is called at the start of a write operation and is used to do anything you need to do in order to open the output file for writing. You are responsible for checking whether such an operation is needed (for example, by setting an appropriate flag to avoid duplicate innitialization). Note that this operation will be called for each channel and you must ensure that any initializations operations are threadsafe if you are not forcing serial channel operations (see force_serial_channel_operations()).

We strongly encourage atomic operations by ensuring that any file handles opened in this function are later closed in close_resources() which will be called at the end of any batch write operation.

Parameters:: channel (int) – the channel for which to initialize the database

abstractmethod MetaWriter._set_output_dtype() → str¶

Purpose: Set the datatype of the data to be saved for each event.

This function returns a string encoding a numpy datatype that tells the writer in what format the event data, which is always in pA, should be stored in the database. Double precision floating point ("<f8") loses no meaningful precision for any input type, and is what the shipped writer uses.

Returns:: A string representing a numpy dtype
Return type:: str

abstractmethod MetaWriter._validate_settings(settings: dict) → None¶

Validate that the settings dict contains the correct information for use by the subclass.

Parameters:: settings (dict) – Parameters required to configure this writer.
Raises:: ValueError – If the settings dict does not contain the correct information.

abstractmethod MetaWriter._write_channel_metadata(channel: int) → None¶

Purpose: Save any metadata required at the level of channels (for example, samplerate).

Given a channel index, write any required metadata for that channel. Typically this is done once per channel on the first related write operation. Remember to close any file handles used either in this function or close_resources() depending on whether you need to keep those resources open for the event writing step that follows.

Parameters:: channel (int) – int indicating which output to flush

abstractmethod MetaWriter._write_data(event: Dict[str, Any], channel: int, index: int, abort: bool | None = False, last_call: bool | None = False) → bool¶

Purpose: Append a single event’s data and metadata to the output file.

Given one event, write it to the active file for channel (appending to an existing file in the case of atomic operations) and return True if that succeeds. If the write fails, raise: raising does not crash poriscope, which carries on to the next event and files the exception’s text as that event’s rejection reason for downstream reporting.

event is the dict MetaEventFinder.get_single_event_data() produces, and is passed through whole rather than exploded into arguments - this took thirteen positional parameters until 2.0.0, of which ten were its keys. Its keys are:

data - the event’s samples, as a 1D numpy array.
start_sample - index of the first sample of the event itself, not of the padding before it, relative to the start of the channel.
padding_before / padding_after - samples of context included on each side. data therefore spans start_sample - padding_before to start_sample + len(event) + padding_after.
baseline_mean / baseline_std - the local baseline the event sits on.

data is always in pA, exactly as the reader’s load_data returns it, so it is consistent with the baseline_mean stored beside it. Until 2.0.0 a writer whose output type matched the reader’s source type was handed unscaled samples with scale and offset keys and a raw_data flag; that path is gone.

Treat a missing key as a programming error and raise, rather than substituting a default: a silently defaulted padding writes an event whose samples do not line up with its own metadata.

Parameters:

event (Dict[str, Any]) – One event, as get_single_event_data builds it.
channel (int) – The channel the event belongs to.
index (int) – The event’s index within that channel.
abort (Optional[bool]) – True to discard the channel’s uncommitted batch and stop.
last_call (Optional[bool]) – True when this is the final event of the channel.

Returns:

True if the event was written.

Return type:

bool