diff --git a/docs/source/_static/css/custom.css b/docs/source/_static/css/custom.css
new file mode 100644
index 00000000..680bd1af
--- /dev/null
+++ b/docs/source/_static/css/custom.css
@@ -0,0 +1,74 @@
+/* arg formatting by line, taken from https://github.com/sphinx-doc/sphinx/issues/1514#issuecomment-742703082 */
+
+/* For general themes */
+div.body, .wy-nav-content {
+ max-width: 1000px; /* Set the content width */
+ margin: 0; /* Remove auto-centering */
+ padding-left: 30; /* Optional: Adjust padding */
+}
+
+/* For Read the Docs theme specifically */
+.wy-nav-content {
+ margin: 0; /* Remove centering (auto) */
+ padding-left: 30px; /* Align content to the left */
+}
+
+
+/*Newlines (\a) and spaces (\20) before each parameter*/
+dl.class em:not([class])::before {
+ content: "\a\20\20\20\20\20\20\20\20\20\20\20\20\20\20\20\20";
+ white-space: pre;
+}
+
+/*Newline after the last parameter (so the closing bracket is on a new line)*/
+dl.class em:not([class]):last-of-type::after {
+ content: "\a";
+ white-space: pre;
+}
+
+/*To have blue background of width of the block (instead of width of content)*/
+dl.class > dt:first-of-type {
+ display: block !important;
+}
+
+.rst-content code.literal, .rst-content tt.literal {
+ color: #2b417e; /* Replace with your desired color */
+}
+.rst-content div[class^=highlight], .rst-content pre.literal-block {
+ margin: 1px 0 14px
+}
+
+.rst-content .section ol li>*, .rst-content .section ul li>*, .rst-content .toctree-wrapper ol li>*, .rst-content .toctree-wrapper ul li>*, .rst-content section ol li>*, .rst-content section ul li>* {
+ margin-top: 0px;
+}
+
+/* Ensure there is 10px spacing between nested list items at different levels*/
+.rst-content li > dl > dt {
+ margin-bottom: 10px;
+}
+.rst-content dd > ul > li {
+ margin-bottom: 10px;
+}
+.rst-content .section ol.simple li>*, .rst-content .section ol.simple li ol, .rst-content .section ol.simple li ul, .rst-content .section ul.simple li>*, .rst-content .section ul.simple li ol, .rst-content .section ul.simple li ul, .rst-content .toctree-wrapper ol.simple li>*, .rst-content .toctree-wrapper ol.simple li ol, .rst-content .toctree-wrapper ol.simple li ul, .rst-content .toctree-wrapper ul.simple li>*, .rst-content .toctree-wrapper ul.simple li ol, .rst-content .toctree-wrapper ul.simple li ul, .rst-content section ol.simple li>*, .rst-content section ol.simple li ol, .rst-content section ol.simple li ul, .rst-content section ul.simple li>*, .rst-content section ul.simple li ol, .rst-content section ul.simple li ul{
+ margin-bottom: 10px;
+}
+
+/* Improve padding and margins for function docstring section titles */
+.rst-content dd > dl > dt {
+ padding-left: 5px;
+ margin-top: 20px;
+ margin-bottom: 10px;
+}
+html.writer-html4 .rst-content dl:not(.docutils) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt, html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt{
+ margin-top: 28px;
+ margin-bottom: 10px;
+}
+
+button.copybtn {
+ height:25px;
+ width:25px;
+ opacity: 0.5;
+ padding: 0;
+ border: none;
+ background: none;
+}
diff --git a/docs/source/_static/html/tutorials/UnitTimes.png b/docs/source/_static/html/tutorials/UnitTimes.png
new file mode 100644
index 00000000..d31f8b55
Binary files /dev/null and b/docs/source/_static/html/tutorials/UnitTimes.png differ
diff --git a/docs/source/_static/html/tutorials/basicUsage.html b/docs/source/_static/html/tutorials/basicUsage.html
new file mode 100644
index 00000000..c9cab5d2
--- /dev/null
+++ b/docs/source/_static/html/tutorials/basicUsage.html
@@ -0,0 +1,431 @@
+
+
Using NWB Data
Using NWB Data
last updated: February 9, 2021
In this tutorial, we demonstrate the reading and usage of the NWB file produced in the File Conversion Tutorial. The output is a near-reproduction of Figure 1e from the Li et al publication, showing raster and peristimulus time histogram (PSTH) plots for neural recordings from anterior lateral motor cortex (ALM). This figure illustrates the main finding of the publication, showing the robustness of motor planning behavior and neural dynamics following short unilateral network silencing via optogenetic inhibition.
Reading NWB Files
NWB files can be read in using the nwbRead() function. This function returns a nwbfile object which is the in-memory representation of the NWB file structure.
nwb = nwbRead('out\ANM255201_20141124.nwb');
Constrained Sets
Analyzed data in NWB is placed under the analysis property, which is a Constrained Set. A constrained set consists of an arbitrary amount of key-value pairs similar to Map containers in MATLAB or a dictionary in Python. However, constrained sets also have the ability to validate their own properties closer to how a typed Object would.
You can get/set values in constrained sets using their respective .get()/.set() methods and retrieve all Set properties using the keys() method, like in a containers.Map.
unit_names = keys(nwb.analysis);
Dynamic Tables
nwb.intervals_trials returns a unique type of table called a Dynamic Table. Dynamic tables inherit from the NWB type types.hdmf_common.DynamicTable and allow for a table-like interface in NWB. In the case below, we grab the special column start_time. Dynamic Tables allow adding your own vectors using the vectordata property, which are Constrained Sets. All columns are represented by either a types.hdmf_common.VectorData or a types.hdmf_common.VectorIndex type.
Data Stubs
The data property of the column id in nwb.units is a types.untyped.DataStub. This object is a representation of a dataset that is not loaded in memory, and is what allows MatNWB to lazily load its file data. To load the data into memory, use the .load() method which extracts all data from the NWB file. Alternatively, you can index into the DataStub directly using conventional MATLAB syntax.
Jagged Arrays in Dynamic Tables
With the new addition of addRow and getRow to Dynamic Tables, the concept of jagged arrays can be worked around and no longer require full understanding outside of specific data format concerns or low-level nwb tool development. The below paragraph is retained in its entirety from its original form as purely informational.
All data in a Dynamic Table must be aligned by row and column, but not all data fits into this paradigm neatly. In order to represent variable amounts of data that is localised to each row and column, NWB uses a concept called Jagged Arrays. These arrays consist of two column types: the familiar types.core.VectorData, and the new types.core.VectorIndex. A Vector Index holds no data, instead holding a reference to another Vector Data and a vector of indices that align to the Dynamic Table dimensions. The indices represent the last index boundary in the Vector Data object for the Vector Index row. As an example, an index of three in the first row of the Vector Index column points to the first three values in the referenced Vector Data column. Subsequently, if the next index were a five, it would indicate the fourth and fifth elements in the referenced Vector Data column.
The jagged arrays serve to represent multiple trials and spike times associated to each unit by id. A convenient way to represent these in MATLAB is to use Map containers where each unit's data is indexed directly by its unit id. Below, we utilize getRow in order to build the same Map.
unit_ids = nwb.units.id.data.load(); % array of unit ids represented within this
% Initialize trials & times Map containers indexed by unit_ids
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/behavior.html b/docs/source/_static/html/tutorials/behavior.html
new file mode 100644
index 00000000..4f70f592
--- /dev/null
+++ b/docs/source/_static/html/tutorials/behavior.html
@@ -0,0 +1,369 @@
+
+Behavior Data
Behavior Data
This tutorial will guide you in writing behavioral data to NWB.
Creating an NWB File
Create an NWBFile object with the required fields (session_description, identifier, and session_start_time) and additional metadata.
nwb = NwbFile( ...
'session_description', 'mouse in open exploration',...
SpatialSeries is a subclass of TimeSeries that represents data in space, such as the spatial direction e.g., of gaze or travel or position of an animal over time.
Create data that corresponds to x, y position over time.
position_data = [linspace(0, 10, 50); linspace(0, 8, 50)]; % 2 x nT array
In SpatialSeries data, the first dimension is always time (in seconds), the second dimension represents the x, y position. However, as described in the dimensionMapNoDataPipes tutorial, when a MATLAB array is exported to HDF5, the array is transposed. Therefore, in order to correctly export the data, in MATLAB the last dimension of an array should be time. SpatialSeries data should be stored as one continuous stream as it is acquired, not by trials as is often reshaped for analysis. Data can be trial-aligned on-the-fly using the trials table. See the trials tutorial for further information.
For position data reference_frame indicates the zero-position, e.g. the 0,0 point might be the bottom-left corner of an enclosure, as viewed from the tracking camera.
To help data analysis and visualization tools know that this SpatialSeries object represents the position of the subject, store the SpatialSeries object inside a Position object, which can hold one or more SpatialSeries objects.
%nwb.processing.set('behavior', behavior_processing_module); % if you have not already added it
BehavioralEvents: Storing behavioral events
BehavioralEvents is an interface for storing behavioral events. We can use it for storing the timing and amount of rewards (e.g. water amount) or lever press times.
reward_amount = [1.0, 1.5, 1.0, 1.5];
event_timestamps = [1.0, 2.0, 5.0, 6.0];
time_series = types.core.TimeSeries( ...
'data', reward_amount, ...
'timestamps', event_timestamps, ...
'description', 'The water amount the subject received as a reward.', ...
%nwb.processing.set('behavior', behavior_processing_module); % if you have not already added it
Storing only the timestamps of the events is possible with the ndx-events NWB extension. You can also add labels associated with the events with this extension. You can find information about installation and example usage here.
BehavioralEpochs: Storing intervals of behavior data
BehavioralEpochs is for storing intervals of behavior data. BehavioralEpochs uses IntervalSeries to represent the time intervals. Create an IntervalSeries object that represents the time intervals when the animal was running. IntervalSeries uses 1 to indicate the beginning of an interval and -1 to indicate the end.
run_intervals = types.core.IntervalSeries( ...
'description', 'Intervals when the animal was running.', ...
Using TimeIntervals to represent time intervals is often preferred over BehavioralEpochs and IntervalSeries. TimeIntervals is a subclass of DynamicTable, which offers flexibility for tabular data by allowing the addition of optional columns which are not defined in the standard DynamicTable class.
sleep_intervals = types.core.TimeIntervals( ...
'description', 'Intervals when the animal was sleeping.', ...
EyeTracking: Storing continuous eye-tracking data of gaze direction
EyeTracking is for storing eye-tracking data which represents direction of gaze as measured by an eye tracking algorithm. An EyeTracking object holds one or more SpatialSeries objects that represent the gaze direction over time extracted from a video.
PupilTracking: Storing continuous eye-tracking data of pupil size
PupilTracking is for storing eye-tracking data which represents pupil size. PupilTracking holds one or more TimeSeries objects that can represent different features such as the dilation of the pupil measured over time by a pupil tracking algorithm.
pupil_diameter = types.core.TimeSeries( ...
'description', 'Pupil diameter extracted from the video of the right eye.', ...
'data', linspace(0.001, 0.002, 50), ...
'starting_time', 1.0, ... % NB: Important to set starting_time when using starting_time_rate
fprintf('Exported NWB file to "%s"\n', 'behavior_tutorial.nwb')
Exported NWB file to "behavior_tutorial.nwb"
+
+
+
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/convertTrials.html b/docs/source/_static/html/tutorials/convertTrials.html
new file mode 100644
index 00000000..1df73a43
--- /dev/null
+++ b/docs/source/_static/html/tutorials/convertTrials.html
@@ -0,0 +1,1119 @@
+
+
+
+
+
+NWB File Conversion Tutorial
+
+
+
+
+
+
+
+
+
NWB File Conversion Tutorial
+
+
How to convert trial-based experimental data to the Neurodata Without Borders file format using MatNWB. This example uses the CRCNS ALM-3 data set. Information on how to download the data can be found on the CRCNS Download Page. One should first familiarize themselves with the file format, which can be found on the ALM-3 About Page under the Documentation files.
+
author: Lawrence Niu
+contact: lawrence@vidriotech.com
+last updated: Sep 14, 2024
The following section describes configuration parameters specific to the publishing script, and can be skipped when implementing your own conversion. The parameters can be changed to fit any of the available sessions.
The animal and session specifier can be changed with the animal and session variable name respectively. metadata_loc, datastructure_loc, and rawdata_loc should refer to the metadata .mat file, the data structure .mat file, and the raw .tar file.
The NWB file will be saved in the output directory indicated by output_directory
+
+
General Information
+
nwb = NwbFile();
+nwb.identifier = identifier;
+nwb.general_source_script = source_script;
+nwb.general_source_script_file_name = source_file;
+nwb.general_lab = 'Svoboda';
+nwb.general_keywords = {'Network models', 'Premotor cortex', 'Short-term memory'};
+nwb.general_institution = ['Janelia Research Campus,'...
+ ' Howard Huges Medical Institute, Ashburn, Virginia 20147, USA'];
+nwb.general_related_publications = ...
+ ['Li N, Daie K, Svoboda K, Druckmann S (2016).',...
+ ' Robust neuronal dynamics in premotor cortex during motor planning.',...
+ ' Nature. 7600:459-64. doi: 10.1038/nature17643'];
+nwb.general_stimulus = 'photostim';
+nwb.general_protocol = 'IACUC';
+nwb.general_surgery = ['Mice were prepared for photoinhibition and ',...
+ 'electrophysiology with a clear-skull cap and a headpost. ',...
+ 'The scalp and periosteum over the dorsal surface of the skull were removed. ',...
+ 'A layer of cyanoacrylate adhesive (Krazy glue, Elmer''s Products Inc.) ',...
+ 'was directly applied to the intact skull. A custom made headpost ',...
+ 'was placed on the skull with its anterior edge aligned with the suture lambda ',...
+ '(approximately over cerebellum) and cemented in place ',...
+ 'with clear dental acrylic (Lang Dental Jet Repair Acrylic; 1223-clear). ',...
+ 'A thin layer of clear dental acrylic was applied over the cyanoacrylate adhesive ',...
+ 'covering the entire exposed skull, ',...
+ 'followed by a thin layer of clear nail polish (Electron Microscopy Sciences, 72180).'];
+nwb.session_description = sprintf('Animal `%s` on Session `%s`', animal, session);
+
+
All properties with the prefix general contain context for the entire experiment such as lab, institution, and experimentors. For session-delimited data from the same experiment, these fields will all be the same. Note that most of this information was pulled from the publishing paper and not from any of the downloadable data.
+
The only required property is the identifier, which distinguishes one session from another within an experiment. In our case, the ALM-3 data uses a combination of session date and animal ID.
+
The ALM-3 File Structure
+
Each ALM-3 session has three files: a metadata .mat file describing the experiment, a data structures .mat file containing analyzed data, and a raw .tar archive containing multiple raw electrophysiology data separated by trials as .mat files. All files will be merged into a single NWB file.
+
Metadata
+
ALM-3 Metadata contains information about the reference times, experimental context, methodology, as well as details of the electrophysiology, optophysiology, and behavioral portions of the experiment. A vast majority of these details are placed in general prefixed properties in NWB.
+
fprintf('Processing Meta Data from `%s`\n', metadata_loc);
+loaded = load(metadata_loc, 'meta_data');
+meta = loaded.meta_data;
+
+% Experiment-specific treatment for animals with the ReaChR gene modification
+isreachr = any(cell2mat(strfind(meta.animalGeneModification, 'ReaChR')));
+
+% Sessions are separated by date of experiment.
+nwb.general_session_id = meta.dateOfExperiment;
+
+% ALM-3 data start time is equivalent to the reference time.
+nwb.session_start_time = datetime([meta.dateOfExperiment meta.timeOfExperiment],...
+ 'InputFormat', 'yyyyMMddHHmmss', 'TimeZone', 'America/New_York'); % Eastern Daylight Time
+nwb.timestamps_reference_time = nwb.session_start_time;
+
+nwb.general_experimenter = strjoin(meta.experimenters, ', ');
+
+
Processing Meta Data from `data/metadata/meta_data_ANM255201_20141124.mat`
+
Ideally, if a raw data field does not correspond directly to a NWB field, one would create their own using a custom NWB extension class. However, since these fields are mostly experimental annotations, we instead pack the extra values into the description field as a string.
+
+% The formatStruct function simply prints the field and values given the struct.
+% An optional cell array of field names specifies whitelist of fields to print.
+% This function is provided with this script in the tutorials directory.
+nwb.general_subject.genotype = formatStruct(...
+ meta, ...
+ {'animalStrain'; 'animalGeneModification'; 'animalGeneCopy';...
+ 'animalGeneticBackground'});
+
+weight = {};
+if ~isempty(meta.weightBefore)
+ weight{end+1} = 'weightBefore';
+end
+if ~isempty(meta.weightAfter)
+ weight{end+1} = 'weightAfter';
+end
+weight = weight(~cellfun('isempty', weight));
+if ~isempty(weight)
+ nwb.general_subject.weight = formatStruct(meta, weight);
+end
+
+% general/experiment_description
+nwb.general_experiment_description = [...
+ formatStruct(meta, {'experimentType'; 'referenceAtlas'}), ...
+ newline, ...
+ formatStruct(meta.behavior, {'task_keyword'})];
+
+% Miscellaneous collection information from ALM-3 that didn't quite fit any NWB
+% properties are stored in general/data_collection.
+nwb.general_data_collection = formatStruct(meta.extracellular,...
+ {'extracellularDataType';'cellType';'identificationMethod';'amplifierRolloff';...
+ 'spikeSorting';'ADunit'});
+
+% Device objects are essentially just a list of device names. We store the probe
+% and laser hardware names here.
+probetype = meta.extracellular.probeType{1};
+probeSource = meta.extracellular.probeSource{1};
+deviceName = [probetype ' (' probeSource ')'];
+nwb.general_devices.set(deviceName, types.core.Device());
+
+if isreachr
+ laserName = 'laser-594nm (Cobolt Inc., Cobolt Mambo 100)';
+else
+ laserName = 'laser-473nm (Laser Quantum, Gem 473)';
+end
+nwb.general_devices.set(laserName, types.core.Device());
+
The NWB ElectrodeGroup object stores experimental information regarding a group of probes. Doing so requires a SoftLink to the probe specified under general_devices. SoftLink objects are direct maps to HDF5 Soft Links on export, and thus, require a true HDF5 path.
+
+% You can specify column names and values as key-value arguments in the DynamicTable
+% constructor.
+dtColNames = {'x', 'y', 'z', 'imp', 'location', 'filtering','group', 'group_name'};
+dynTable = types.hdmf_common.DynamicTable(...
+ 'colnames', dtColNames,...
+ 'description', 'Electrodes',...
+ 'x', types.hdmf_common.VectorData('description', 'x coordinate of the channel location in the brain (+x is posterior).'),...
+ 'y', types.hdmf_common.VectorData('description', 'y coordinate of the channel location in the brain (+y is inferior).'),...
+ 'z', types.hdmf_common.VectorData('description', 'z coordinate of the channel location in the brain (+z is right).'),...
+ 'imp', types.hdmf_common.VectorData('description', 'Impedance of the channel.'),...
+ 'location', types.hdmf_common.VectorData('description', ['Location of the electrode (channel). '...
+ 'Specify the area, layer, comments on estimation of area/layer, stereotaxic coordinates if '...
+ 'in vivo, etc. Use standard atlas names for anatomical regions when possible.']),...
+ 'filtering', types.hdmf_common.VectorData('description', 'Description of hardware filtering.'),...
+ 'group', types.hdmf_common.VectorData('description', 'Reference to the ElectrodeGroup this electrode is a part of.'),...
+ 'group_name', types.hdmf_common.VectorData('description', 'Name of the ElectrodeGroup this electrode is a part of.'));
+
+% Raw HDF5 path to the above electrode group. Referenced by
+% the general/extracellular_ephys Dynamic Table
+egroupPath = ['/general/extracellular_ephys/' deviceName];
+eGroupReference = types.untyped.ObjectView(egroupPath);
+for i = 1:length(meta.extracellular.siteLocations)
+ location = meta.extracellular.siteLocations{i};
+ % Add each row in the dynamic table. The `id` column is populated
+ % dynamically.
+ dynTable.addRow(...
+ 'x', location(1), 'y', location(2), 'z', location(3),...
+ 'imp', 0,...
+ 'location', recordingLocation,...
+ 'filtering', '',...
+ 'group', eGroupReference,...
+ 'group_name', probetype);
+end
+
+
The group column in the Dynamic Table contains an ObjectView to the previously created ElectrodeGroup. An ObjectView can be best thought of as a direct pointer to another typed object. It also directly maps to a HDF5 Object Reference, thus the HDF5 path requirement. ObjectViews are slightly different from SoftLinks in that they can be stored in datasets (data columns, tables, and data fields in NWBData objects).
The electrodes property in extracellular_ephys is a special keyword in NWB that must be paired with a Dynamic Table. These are tables which can have an unbounded number of columns and rows, each as their own dataset. With the exception of the id column, all other columns must be VectorData or VectorIndex objects. The id column, meanwhile, must be an ElementIdentifiers object. The names of all used columns are specified in the in the colnames property as a cell array of strings.
The ALM-3 data structures .mat file contains analyzed spike data, trial-specific parameters, and behavioral analysis data.
+
Hashes
+
ALM-3 stores its data structures in the form of hashes which are essentially the same as python's dictionaries or MATLAB's maps but where the keys and values are stored under separate struct fields. Getting a hashed value from a key involves retrieving the array index that the key is in and applying it to the parallel array in the values field.
+
You can find more information about hashes and how they're used on the ALM-3 about page.
+
fprintf('Processing Data Structure `%s`\n', datastructure_loc);
+loaded = load(datastructure_loc, 'obj');
+data = loaded.obj;
+
+% wherein each cell is one trial. We must populate this way because trials
+% may not be in trial order.
+% Trial timeseries will be a compound type under intervals/trials.
+trial_timeseries = cell(size(data.trialIds));
+
+
Processing Data Structure `data/data_structure_files/data_structure_ANM255201_20141124.mat`
+
+
NWB comes with default support for trial-based data. These must be TimeIntervals that are placed in the intervals property. Note that trials is a special keyword that is required for PyNWB compatibility.
The timeseries property of the TimeIntervals object is an example of a compound data type. These types are essentially tables of data in HDF5 and can be represented by a MATLAB table, an array of structs, or a struct of arrays. Beware: validation of column lengths here is not guaranteed by the type checker until export.
+
+VectorIndex objects index into a larger VectorData column. The object that is being referenced is indicated by the target property, which uses an ObjectView. Each element in the VectorIndex marks the last element in the corresponding vector data object for the VectorIndex row. Thus, the starting index for this row would be the previous index + 1. Note that these indices must be 0-indexed for compatibility with pynwb. You can see this in effect with the timeseries property which is indexed by the timeseries_index property.
+
Though TimeIntervals is a subclass of the DynamicTable type, we opt for populating the Dynamic Table data by column instead of using `addRow` here because of how the data is formatted. DynamicTable is flexible enough to accomodate both styles of data conversion.
Ephus spike data is separated into units which directly maps to the NWB property of the same name. Each such unit contains a group of analysed waveforms and spike times, all linked to a different subset of trials IDs.
+
The waveforms are placed in the analysis Set and are paired with their unit name ('unitx' where 'x' is some unit ID).
+
Trial IDs, wherever they are used, are placed in a relevent control property in the data object and will indicate what data is associated with what trial as defined in trials's id column.
To better understand how spike_times_index and spike_times map to each other, refer to this diagram from the Extracellular Electrophysiology Tutorial.
+
Raw Acquisition Data
+
Each ALM-3 session is associated with a large number of raw voltage data grouped by trial ID. To map this data to NWB, each trial is created as its own ElectricalSeries object under the name 'trial n' where 'n' is the trial ID. The trials are then linked to the trials dynamic table for easy referencing.
+
fprintf('Processing Raw Acquisition Data from `%s` (will take a while)\n', rawdata_loc);
+untarLoc = strrep(rawdata_loc, '.tar', '');
+if ~isfolder(untarLoc)
+ untar(rawdata_loc, fileparts(rawdata_loc));
+end
+
+rawfiles = dir(untarLoc);
+rawfiles = fullfile(untarLoc, {rawfiles(~[rawfiles.isdir]).name});
+
+nrows = length(nwb.general_extracellular_ephys_electrodes.id.data);
+tablereg = types.hdmf_common.DynamicTableRegion(...
+ 'description','Relevent Electrodes for this Electrical Series',...
+ 'table',types.untyped.ObjectView('/general/extracellular_ephys/electrodes'),...
+ 'data',(1:nrows) - 1);
+objrefs = cell(size(rawfiles));
+
+endTimestamps = trials_epoch.start_time.data;
+for i=1:length(rawfiles)
+ tnumstr = regexp(rawfiles{i}, '_trial_(\d+)\.mat$', 'tokens', 'once');
+ tnumstr = tnumstr{1};
+ rawdata = load(rawfiles{i}, 'ch_MUA', 'TimeStamps');
+ tnum = str2double(tnumstr);
+
+ if tnum > length(endTimestamps)
+ continue; % sometimes there are extra trials without an associated start time.
+ end
+
+ es = types.core.ElectricalSeries(...
+ 'data', rawdata.ch_MUA,...
+ 'description', ['Raw Voltage Acquisition for trial ' tnumstr],...
+ 'electrodes', tablereg,...
+ 'timestamps', rawdata.TimeStamps);
+ tname = ['trial ' tnumstr];
+ nwb.acquisition.set(tname, es);
+
+ endTimestamps(tnum) = endTimestamps(tnum) + rawdata.TimeStamps(end);
+ objrefs{tnum} = types.untyped.ObjectView(['/acquisition/' tname]);
+end
+
+% Link to the raw data by adding the acquisition column with ObjectViews
+% to the data
+emptyrefs = cellfun('isempty', objrefs);
+objrefs(emptyrefs) = {types.untyped.ObjectView('')};
+
+trials_epoch.addColumn('acquisition', types.hdmf_common.VectorData(...
+ 'description', 'soft link to acquisition data for this trial',...
+ 'data', [objrefs{:}]'));
+
+trials_epoch.stop_time = types.hdmf_common.VectorData(...
+ 'data', endTimestamps',...
+ 'description', 'the end time of each trial');
+trials_epoch.colnames{end+1} = 'stop_time';
+
+
Processing Raw Acquisition Data from `data/RawVoltageTraces/ANM255201_20141124.tar` (will take a while)
+
+
Add timeseries to trials_epoch
+
First, we'll format and store trial_timeseries into intervals_trials. note that timeseries_index data is 0-indexed.
+
ts_len = cellfun('size', trial_timeseries, 1);
+nwb.intervals_trials.timeseries_index = types.hdmf_common.VectorIndex(...
+ 'description', 'Index into Timeseries VectorData', ...
+ 'data', cumsum(ts_len)', ...
+ 'target', types.untyped.ObjectView('/intervals/trials/timeseries') );
+
+% Intervals/trials/timeseries is a compound type so we use cell2table to
+% convert this 2-d cell array into a compatible table.
+is_len_nonzero = ts_len > 0;
+trial_timeseries_table = cell2table(vertcat(trial_timeseries{is_len_nonzero}),...
+ 'VariableNames', {'timeseries', 'idx_start', 'count'});
+trial_timeseries_table = movevars(trial_timeseries_table, 'timeseries', 'After', 'count');
+
+interval_trials_timeseries = types.core.TimeSeriesReferenceVectorData(...
+ 'description', 'Index into TimeSeries data', ...
+ 'data', trial_timeseries_table);
+nwb.intervals_trials.timeseries = interval_trials_timeseries;
+nwb.intervals_trials.colnames{end+1} = 'timeseries';
+
Neurophysiology data can be quite large, often in the 10s of GB per session and sometimes much larger. Here, we demonstrate methods in MatNWB that allow you to deal with large datasets. These methods are compression and iterative write. Both of these techniques use the types.untyped.DataPipe object, which sends specific instructions to the HDF5 backend about how to store data.
+
Compression - basic implementation
+
To compress experimental data (in this case a 3D matrix with dimensions [250 250 70]) one must assign it as a DataPipe type:
This is the most basic way to acheive compression, and all of the optimization decisions are automatically determined by MatNWB.
+
Background
+
HDF5 has built-in ability to compress and decompress individual datasets. If applied intelligently, this can dramatically reduce the amount of space used on the hard drive to represent the data. The end user does not need to worry about the compression status of the dataset- HDF5 will automatically decompress the dataset on read.
+
The above example uses default chunk size and compression level (3). To optimize compression, compressionLevel and chunkSize must be considered. compressionLevel ranges from 0 - 9 where 9 is the highest level of compression and 0 is the lowest. chunkSize is less intuitive to adjust; to implement compression, chunk size must be less than data size.
+
+DataPipe Arguments
+
+
+
maxSize
Sets the maximum size of the HDF5 Dataset. Unless using iterative writing, this should match the size of Data. To append data later, use the maxSize for the full dataset. You can use Inf for a value of a dimension if you do not know its final size.
+
data
The data to compress. Must be numerical data.
+
axis
Set which axis to increment when appending more data.
+
dataType
Sets the type of the experimental data. This must be a numeric data type. Useful to include when using iterative write to append data as the appended data must be the same data type. If data is provided and dataType is not, the dataType is inferred from the provided data.
+
chunkSize
Sets chunk size for the compression. Must be less than maxSize.
+
compressionLevel
Level of compression ranging from 0-9 where 9 is the highest level of compression. The default is level 3.
+
offset
Axis offset of dataset to append. May be used to overwrite data.
+
+
Chunking
+
HDF5 Datasets can be either stored in continuous or chunked mode. Continuous means that all of the data is written to one continuous block on the hard drive, and chunked means that the dataset is automatically split into chunks that are distributed across the hard drive. The user does not need to know the mode used- HDF5 handles the gathering of chunks automatically. However, it is worth understanding these chunks because they can have a big impact on space used and read and write speed. When using compression, the dataset MUST be chunked. HDF5 is not able to apply compression to continuous datasets.
+
If chunkSize is not explicitly specified, dataPipe will determine an appropriate chunk size. However, you can optimize the performance of the compression by manually specifying the chunk size using chunkSize argument.
+
We can demonstrate the benefit of chunking by exploring the following scenario. The following code utilizes DataPipe's default chunk size:
This results in a file size of 47MB (too large), and the process takes 11 seconds (far too long). Setting the chunk size manually as in the example code below resolves these issues:
This change results in the operation completing in 0.7 seconds and resulting file size of 1.1MB. The chunk size was chosen such that it spans each individual row of the matrix.
+
Use the combination of arugments that fit your need. When dealing with large datasets, you may want to use iterative write to ensure that you stay within the bounds of your system memory and use chunking and compression to optimize storage, read and write of the data.
+
Iterative Writing
+
If experimental data is close to, or exceeds the available system memory, performance issues may arise. To combat this effect of large data, DataPipe can utilize iterative writing, where only a portion of the data is first compressed and saved, and then additional portions are appended.
+
To demonstrate, we can create a nwb file with a compressed time series data:
+
dataPart1 = randi(250, 1, 1000); % "load" 1/4 of the entire dataset
+fullDataSize = [1 40000]; % this is the size of the TOTAL dataset
+
+% create an nwb structure with required fields
+nwb=NwbFile( ...
+ 'session_start_time', datetime('2020-01-01 00:00:00', 'TimeZone', 'local'), ...
+ 'identifier', 'ident1', ...
+ 'session_description', 'DataPipeTutorial');
+
+% compress the data
+fData_use = types.untyped.DataPipe( ...
+ 'data', dataPart1, ...
+ 'maxSize', fullDataSize, ...
+ 'axis', 2);
+
+%Set the compressed data as a time series
+fdataNWB = types.core.TimeSeries( ...
+ 'data', fData_use, ...
+ 'data_unit', 'mV', ...
+ 'starting_time', 0.0, ...
+ 'starting_time_rate', 30.0);
+
+nwb.acquisition.set('time_series', fdataNWB);
+
+nwbExport(nwb, 'DataPipeTutorial_iterate.nwb');
+
+
To append the rest of the data, simply load the NWB file and use the append method:
+
nwb = nwbRead('DataPipeTutorial_iterate.nwb', 'ignorecache'); %load the nwb file with partial data
+
+% "load" each of the remaining 1/4ths of the large dataset
+for i = 2:4 % iterating through parts of data
+ dataPart_i=randi(250, 1, 10000); % faked data chunk as if it was loaded
+ nwb.acquisition.get('time_series').data.append(dataPart_i); % append the loaded data
+end
+
+
The axis property defines the dimension in which additional data will be appended. In the above example, the resulting dataset will be 4000x1. However, if we set axis to 2 (and change fullDataSize appropriately), then the resulting dataset will be 1000x4.
+
Timeseries example
+
Following is an example of how to compress and add a timeseries to an NWB file:
This tutorial demonstrates how the dimensions of a MATLAB array maps onto a dataset in HDF5. There are two main differences between the way MATLAB and HDF5 represents dimensions:
C-ordering vs F-ordering: HDF5 is C-ordered, which means it stores data in a rows-first pattern, whereas MATLAB is F-ordered, storing data in the reverse pattern, with the last dimension of the array stored consecutively. The result is that the data in HDF5 is effectively the transpose of the array in MATLAB.
1D data (i.e vectors): HDF5 can store 1-D arrays, but in MATLAB the lowest dimensionality of an array is 2-D.
Due to differences in how MATLAB and HDF5 represent data, the dimensions of datasets are flipped when writing to/from file in MatNWB. Additionally, MATLAB represents 1D vectors in a 2D format, either as row vectors or column vectors, whereas HDF5 treats vectors as truly 1D. Consequently, when a 1D dataset from HDF5 is loaded into MATLAB, it is always represented as a column vector. To avoid unintentional changes in data dimensions, it is therefore recommended to avoid writing row vectors into an NWB file for 1D datasets.
Contrast this tutorial with the dimensionMapWithDataPipestutorial that illustrates how vectors are represented differently when using DataPipe objects within VectorData objects.
You can examine the dimensions of the datasets on file using HDFView. Screenshots for this file are below.
+
+
+
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/dimensionMapWithDataPipes.html b/docs/source/_static/html/tutorials/dimensionMapWithDataPipes.html
new file mode 100644
index 00000000..fb17536d
--- /dev/null
+++ b/docs/source/_static/html/tutorials/dimensionMapWithDataPipes.html
@@ -0,0 +1,110 @@
+
+MatNWB <-> HDF5 Dimension Mapping
MatNWB <-> HDF5 Dimension Mapping
This tutorial is easier to follow if you have already looked at the dimensionMapNoDataPipes tutorial or if you compare these side by side.
The key difference when using DataPipe instead of VectorData is that 1D data can be represented in HDF5 as 2D, thus allowing you to write either row or column vectors. This is made possible because of the maxSize property of the DataPipe class, which lets you specify a max size for each dimension. By setting the maxSize to [1, N] or [N, 1], vectors in HDF5 are represented as 2D arrays, just like in MATLAB. The flipping of the dimension order still applies, so a row vector in MATLAB becomes a column vector in HDF5 and vice versa.
Please note: The following tutorial mixes row and column vectors and does not produce a valid dynamic table. The tutorial is only meant to showcase how data maps onto HDF5 datasets when using DataPipe objects.
Create Table
First, create an expandable TimeIntervals table of height 10.
You can examine the dimensions of the datasets on file using HDFView. Screenshots for this file are below.
+
+
+
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/dynamic_tables.html b/docs/source/_static/html/tutorials/dynamic_tables.html
new file mode 100644
index 00000000..af3f6c88
--- /dev/null
+++ b/docs/source/_static/html/tutorials/dynamic_tables.html
@@ -0,0 +1,577 @@
+
+DynamicTables Tutorial
DynamicTables Tutorial
This is a user guide to interacting with DynamicTable objects in MatNWB.
Start by setting up your MATLAB workspace. The code below adds the directory containing the MatNWB package to the MATLAB search path. MatNWB works by automatically creating API classes based on a defined schema.
%{
path_to_matnwb = '~/Repositories/matnwb'; % change to your own path location
addpath(genpath(pwd));
%}
Constructing a table with initialized columns
The DynamicTableclass represents a column-based table to which you can add custom columns. It consists of a description, a list of columns , and a list of row IDs. You can create a DynamicTable by first defining the VectorData objects that will make up the columns of the table. Each VectorData object must contain the same number of rows. A list of rows IDs may be passed to the DynamicTable using the id argument. Row IDs are a useful way to access row information independent of row location index. The list of row IDs must be cast as an ElementIdentifiers object before being passed to the DynamicTable object. If no value is passed to id, an ElementIdentifiers object with 0-indexed row IDs will be created for you automatically.
MATLAB Syntax Note: Using column vectors is crucial to properly build vectors and tables. When defining individual values, make sure to use semi-colon (;) instead of instead of comma (,) when defining the data fields of these.
col1 = types.hdmf_common.VectorData( ...
'description', 'column #1', ...
'data', [1;2] ...
);
col2 = types.hdmf_common.VectorData( ...
'description', 'column #2', ...
'data', {'a';'b'} ...
);
my_table = types.hdmf_common.DynamicTable( ...
'description', 'an example table', ...
'colnames', {'col1', 'col2'}, ...
'col1', col1, ...
'col2', col2, ...
'id', types.hdmf_common.ElementIdentifiers('data', [0;1]) ... % 0-indexed, for compatibility with Python
You can add rows to an existing DynamicTable using the object's addRow method. One way of using this method is to pass in the names of columns as parameter names followed by the elements to append. The class of the elements of the column must match the elements to append.
You can add new columns to an existing DynamicTable object using the addColumn method. One way of using this method is to pass in the names of each new column followed by the corresponding values for each new column. The height of the new columns must match the height of the table.
As an alternative to building a dynamic table using the DynamicTable and VectorData data types, it is also possible to create a MATLAB table and convert it to a dynamic table. Lets create the same table as before, but using MATLAB's table class:
% Create a table with two variables (columns):
T = table([1;2], {'a';'b'}, 'VariableNames', {'col1', 'col2'});
dynamic_table = util.table2nwb(T, 'A MATLAB table that was converted to a dynamic table')
dynamic_table =
DynamicTable with properties:
+
+ id: [1×1 types.hdmf_common.ElementIdentifiers]
+ colnames: {'col1' 'col2' 'col3' 'col4'}
+ description: 'A MATLAB table that was converted to a dynamic table'
+ vectordata: [4×1 types.untyped.Set]
+
Enumerated (categorical) data
EnumData is a special type of column for storing an enumerated data type. This way each unique value is stored once, and the data references those values by index. Using this method is more efficient than storing a single value many times, and has the advantage of communicating to downstream tools that the data is categorical in nature.
Warning Regarding EnumData
EnumDatais currently an experimental feature and as such should not be used in a production environment.
MyTable = types.hdmf_common.DynamicTable('description', 'an example table');
MyTable.vectordata.set('cell_type_elements', CellTypeElements); % the *_elements format is required for compatibility with pynwb
MyTable.addColumn('cell_type', CellType);
Ragged array columns
A table column with a different number of elements for each row is called a "ragged array column." To define a table with a ragged array column, pass both the VectorData and the corresponding VectorIndex as columns of the DynamicTable object. The VectorData columns will contain the data values. The VectorIndex column serves to indicate how to arrange the data across rows. By convention the VectorIndex object corresponding to a particular column must have have the same name with the addition of the '_index' suffix.
Below, the VectorIndex values indicate to place the 1st to 3rd (inclusive) elements of the VectorData into the first row and 4th element into the second row. The resulting table will have the cell {'1a'; '1b'; '1c'} in the first row and the cell {'2a'} in the second row.
col1 = types.hdmf_common.VectorData( ...
'description', 'column #1', ...
'data', {'1a'; '1b'; '1c'; '2a'} ...
);
col1_index = types.hdmf_common.VectorIndex( ...
'description', 'column #1 index', ...
'target',types.untyped.ObjectView(col1), ... % object view of target column
'id', types.hdmf_common.ElementIdentifiers('data', [0; 1]) ... % 0-indexed, for compatibility with Python
);
Adding ragged array rows
You can add a new row to the ragged array column. Under the hood, the addRow method will add the appropriate value to the VectorIndex column to maintain proper formatting.
You can access data from entire rows of a DynamicTable object by calling the getRow method for the corresponding object. You can supply either an individual row number or a list of row numbers.
my_table.getRow(1)
ans = 1×4 table
col1
col2
col3
col4
1
1
'a'
100
'a1'
If you want to access values for just a subset of columns you can pass in the 'columns' argument along with a cell array with the desired column names
my_table.getRow(1:3, 'columns', {'col1'})
ans = 3×1 table
col1
1
1
2
2
3
3
You can also access specific rows by their corresponding row ID's, if they have been defined, by supplying a 'true' Boolean to the 'useId' parameter
my_table.getRow(1, 'useId', true)
ans = 1×4 table
col1
col2
col3
col4
1
2
'b'
200
'b2'
For a ragged array columns, the getRow method will return a cell with different number of elements for each row
table_ragged_col.getRow(1:2)
ans = 2×1 table
col1
1
[{'1a'};{'1b'};{'1c'}]
2
1×1 cell
Accessing column elements
To access all rows from a particular column use the .get method on the vectordata field of the DynamicTable object
my_table.vectordata.get('col2').data
ans = 3×1 cell
'a' 'b' 'c'
Referencing rows of other tables
You can create a column that references rows of other tables by adding a DynamicTableRegion object as a column of a DynamicTable. This is analogous to a foreign key in a relational database. The DynamicTableRegion class takes in an ObjectView object as argument. ObjectView objects create links from one object type referencing another.
You can convert a DynamicTableobject to a MATLAB table by making use of the object's toTable method. This is a useful way to view the whole table in a human-readable format.
my_table.toTable()
ans = 3×5 table
id
col1
col2
col3
col4
1
0
1
'a'
100
'a1'
2
1
2
'b'
200
'b2'
3
2
3
'c'
300
'c3'
When the DynamicTableobject contains a column that references other tables, you can pass in a Boolean to indicate whether to include just the row indices of the referenced table. Passing in false will result in inclusion of the referenced rows as nested tables.
dtr_table.toTable(false)
ans = 4×3 table
id
data_col
dtr_col
1
0
'a'
1×4 table
2
1
'b'
1×4 table
3
2
'c'
1×4 table
4
3
'd'
1×4 table
Creating an expandable table
When using the default HDF5 backend, each column of these tables is an HDF5 Dataset, which by default are set to an unchangeable size. This means that once a file is written, it is not possible to add a new row. If you want to be able to save this file, load it, and add more rows to the table, you will need to set this up when you create the VectorData andElementIdentifierscolumns of aDynamicTable. Specifically, you must wrap the column data with a DataPipe object. The DataPipe class takes in maxSize and axis as arguments to indicate the maximum desired size for each axis and the axis to which to append to, respectively. For example, creating a DataPipe object with a maxSize value equal to [Inf, 1] indicates that the number of rows may increase indifinetely. In contrast, setting maxSize equal to [8, 1] would allow the column to grow to a maximum height of 8.
Note: DataPipe objects change how the dimension of the datasets for each column map onto the shape of HDF5 datasets. See README for more details.
Multidimensional Columns
The order of dimensions of multidimensional columns in MatNWB is reversed relative to the Python HDMF package (see README for detailed explanation). Therefore, the height of a multidimensional column belonging to a DynamicTableobject is defined by the shape of its last dimension. A valid DynamicTablemust have matched height across columns.
'id', types.hdmf_common.ElementIdentifiers('data', [0; 1; 2]) ... % 0-indexed, for compatibility with Python
);
Adding rows to multidimensional array columns
DynamicTableobjects with multidimensional array columns can also be constructed by adding a single row at a time. This method makes use of DataPipe objects due to the fact that MATLAB doesn't support singleton dimensions for arrays with more than 2 dimensions. The code block below demonstrates how to build a DynamicTableobject with a mutidimensional raaged array column in this manner.
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/dynamically_loaded_filters.html b/docs/source/_static/html/tutorials/dynamically_loaded_filters.html
new file mode 100644
index 00000000..7250e315
--- /dev/null
+++ b/docs/source/_static/html/tutorials/dynamically_loaded_filters.html
@@ -0,0 +1,141 @@
+
+Using Dynamically Loaded Filters in MatMWB
Using Dynamically Loaded Filters in MatMWB
Installing Dynamically Loaded Filters
HDF5 can use various filters to compress data when writing datasets. GZIP is the default filter, and it can be read with any HDF5 installation without any setup, but many users find that other filters, such as Zstd, offer better performance. If you want to read an HDF5 Dataset that was compressed using another filter in MATLAB, such as Zstd, you will need to configure MATLAB to read using dynamically loaded filters.
The easiest way we have found to set up dynamically loaded filters is to use the Python package hdf5plugin. This library has a sophisticated installation process that compiles several of the most popular dynamically loaded filters and works across popular operating systems. Installing this Python package is a trick that allows us to offload the tricky parts of installing dynamically loaded filters in MATLAB.
Linux or Mac
1. In your Terminal window, install hdf5plugin:
pip install hdf5plugin
2. In that same Terminal window, set the environment variable HDF5_PLUGIN_PATH:
2. Set the environment variable HDF5_PLUGIN_PATH to point to the local installation of the plugins (from hdf5plugin.PLUGINS_PATH) through System Properties > Advanced > Environment Variables:
3. Restart MATLAB.
That's it! Now you can read datasets that use the following filters:
Bitshuffle
Blosc
FciDecomp
LZ4
Zfp
Zstd
The beauty of HDF5 is that it handles the rest under the hood. When you read a dataset that uses any of these filters, HDF5 will identify the correct decompression algorithm and decompress the data on-the-fly as you access it from the dataset.
For more information about installing filter plugins, see the MATLAB documentation.
Writing with Dynamically Loaded Filters
To write with dynamically loaded filters, first follow the installation steps above. This feature requires MATLAB version ≥ 2022a.
DataPipe objects can be used to write using Dynamically loaded filters. This tutorial will be using the Zstd dynamic filter as an example.
The DynamicFilter property takes in an enumerated type Filter which is a hard-coded list of all listed registered filter plugins in HDF5.
Some filter plugins allow for setting special configuration parameters to modify the filter's behavior. The DynamicFilter property type contains a modifiable parameters field which can be used to set your parameters. This is equivalent to setting the cd_values argument in HDF5. In the case of the Zstandard HDF5 plugin, the first (and only) array argument value indicates the compression level.
zstdProperty.parameters = 4; % compression level.
Multiple Filters
You can use multiple dynamic filters by concatenating multiple DynamicFilter properties together. They will be applied in order of the inserted array.
The DataPipe class takes in a keyword argument called filters which is an array of DynamicFilter objects. Supplying a 'filters' argument will deactivate the default GZIP compression.
% We're already compressing using zstd so we should disable
The data is now compressed using Zstandard compression using a compression level of 4 and Shuffled
+
+
+
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/ecephys.html b/docs/source/_static/html/tutorials/ecephys.html
new file mode 100644
index 00000000..8d82796e
--- /dev/null
+++ b/docs/source/_static/html/tutorials/ecephys.html
@@ -0,0 +1,1356 @@
+
+Neurodata Without Borders Extracellular Electrophysiology Tutorial
Neurodata Without Borders Extracellular Electrophysiology Tutorial
Create fake data for a hypothetical extracellular electrophysiology experiment. The types of data we will convert are:
Voltage recording
Local field potential (LFP)
Spike times
It is recommended to first work through the Introduction to MatNWB tutorial, which demonstrates installing MatNWB and creating an NWB file with subject information, animal position, and trials, as well as writing and reading NWB files in MATLAB.
Setting up the NWB File
An NWB file represents a single session of an experiment. Each file must have a session_description, identifier, and session start time. Create a new NWBFile object with those and additional metadata. For all MatNWB functions, we use the Matlab method of entering keyword argument pairs, where arguments are entered as name followed by value.
nwb = NwbFile( ...
'session_description', 'mouse in open exploration',...
In order to store extracellular electrophysiology data, you first must create an electrodes table describing the electrodes that generated this data. Extracellular electrodes are stored in an electrodes table, which is also a DynamicTable. electrodes has several required fields: x, y, z, impedance, location, filtering, and electrode_group.
Electrodes Table
Since this is a DynamicTable, we can add additional metadata fields. We will be adding a "label" column to the table.
In the above loop, we createElectrodeGroup objects. Theelectrodes table then uses anObjectView in each row to link to the correspondingElectrodeGroup object. AnObjectView is an object that allow you to create a link from one neurodata type referencing another.
ElectricalSeries
Voltage data are stored inElectricalSeriesobjects.ElectricalSeriesis a subclass ofTimeSeriesspecialized for voltage data. In order to create ourElectricalSeriesobject, we will need to reference a set of rows in theelectrodestable to indicate which electrodes were recorded. We will do this by creating aDynamicTableRegion, which is a type of link that allows you to reference specific rows of aDynamicTable, such as theelectrodestable, by row indices.
Create aDynamicTableRegionthat references all rows of theelectrodestable.
Local field potential (LFP) refers in this case to data that has been downsampled and/or filtered from the original acquisition data and is used to analyze signals in the lower frequency range. Filtered and downsampled LFP data would also be stored in an ElectricalSeries. To help data analysis and visualization tools know that this ElectricalSeries object represents LFP data, store it inside an LFP object, then place the LFP object in a ProcessingModule named 'ecephys'. This is analogous to how we stored the SpatialSeries object inside of a Position object and stored the Position object in a ProcessingModule named 'behavior' earlier.
Spike times are stored in another DynamicTable of subtypeUnits. The defaultUnits table is at/units in the HDF5 file. You can add columns to theUnits table just like you did forelectrodes andtrials. Here, we generate some random spike data and populate the table.
num_cells = 10;
firing_rate = 20;
spikes = cell(1, num_cells);
for iShank = 1:num_cells
spikes{iShank} = rand(1, randi([16, 28]));
end
spikes
spikes = 1×10 cell
1
2
3
4
5
6
7
8
9
10
1
1×21 double
1×24 double
1×18 double
1×28 double
1×25 double
1×18 double
1×21 double
1×28 double
1×16 double
1×19 double
Spike times are an example of a ragged array- it's like a matrix, but each row has a different number of elements. We can represent this type of data as an indexed column of the units DynamicTable. These indexed columns have two components, the vector data object that holds the data and the vector index object that holds the indices in the vector that indicate the row breaks. You can use the convenience functionutil.create_indexed_column to create these objects.
In MATLAB, while the Units table is used to store spike times and waveform data for spike-sorted, single-unit activity, you may also want to store spike times and waveform snippets of unsorted spiking activity. This is useful for recording multi-unit activity detected via threshold crossings during data acquisition. Such information can be stored using SpikeEventSeries objects.
% In the SpikeEventSeries the dimensions should be ordered as
As mentioned above,ElectricalSeries objects are meant for storing specific types of extracellular recordings. In addition to this TimeSeries class, NWB provides some Processing Modules for designating the type of data you are storing. We will briefly discuss them here, and refer the reader to the API documentation and Intro to NWB for more details on using these objects.
For storing unsorted spiking data, there are two options. Which one you choose depends on what data you have available. If you need to store complete and/or continuous raw voltage traces, you should store the traces with ElectricalSeries objects as acquisition data, and use the EventDetection class for identifying the spike events in your raw traces. If you do not want to store the raw voltage traces and only the waveform ‘snippets’ surrounding spike events, you should use SpikeEventSeries objects.
The results of spike sorting (or clustering) should be stored in the top-level Units table. The Units table can hold just the spike times of sorted units or, optionally, include additional waveform information. You can use the optional predefined columns waveform_mean, waveform_sd, and waveforms in the Units table to store individual and mean waveform data.
For local field potential data, there are two options. Again, which one you choose depends on what data you have available. With both options, you should store your traces withElectricalSeriesobjects. If you are storing unfiltered local field potential data, you should store theElectricalSeries objects in LFP data interface object(s). If you have filtered LFP data, you should store the ElectricalSeries objects inFilteredEphys data interface object(s).
Writing the NWB File
nwbExport(nwb, 'ecephys_tutorial.nwb')
Reading NWB Data
Data arrays are read passively from the file. CallingTimeSeries.data does not read the data values, but presents an HDF5 object that can be indexed to read data. This allows you to conveniently work with datasets that are too large to fit in RAM all at once. load with no input arguments reads the entire dataset:
If all you need is a data region, you can index a DataStub object like you would any normal array in MATLAB, as shown below. When indexing the dataset this way, only the selected region is read from disk into RAM. This allows you to handle very large datasets that would not fit entirely into RAM.
The following tutorial describes storage of intracellular electrophysiology data in NWB. NWB supports storage of the time series describing the stimulus and response, information about the electrode and device used, as well as metadata about the organization of the experiment.
Illustration of the hierarchy of metadata tables used to describe the organization of intracellular electrophysiology experiments.
Creating an NWBFile
When creating an NWB file, the first step is to create theNWBFile, which you can create using the NwbFile command.
Intracellular electrode metadata is represented byIntracellularElectrodeobjects. Create an electrode object, which requires a link to the device of the previous step. Then add it to the NWB file.
Intracellular stimulus and response data are represented with subclasses ofPatchClampSeries. A stimulus is described by a time series representing voltage or current stimulation with a particular set of parameters. There are two classes for representing stimulus data:
The response is then described by a time series representing voltage or current recorded from a single cell using a single intracellular electrode via one of the following classes:
'description', 'category table for lab-specific recording metadata', ...
'colnames', {'location'}, ...
'id', types.hdmf_common.ElementIdentifiers( ...
'data', int64([0, 1, 2]) ...
), ...
'location', types.hdmf_common.VectorData( ...
'data', {'Mordor', 'Gondor', 'Rohan'}, ...
'description', 'Recording location in Middle Earth' ...
) ...
) ...
);
In an AlignedDynamicTable all category tables must align with the main table, i.e., all tables must have the same number of rows and rows are expected to correspond to each other by index.
We can also add custom columns to any of the subcategory tables, i.e., the electrodes, stimuli, and responses tables, and any custom subcategory tables. All we need to do is indicate the name of the category we want to add the column to.
% Add voltage threshold as column of electrodes table
To describe the organization of intracellular experiments, the metadata is organized hierarchically in a sequence of tables. All of the tables are so-called DynamicTables enabling users to add columns for custom metadata. Storing data in hierarchical tables has the advantage that it allows us to avoid duplication of metadata. E.g., for a single experiment we only need to describe the metadata that is constant across an experimental condition as a single row in the SimultaneousRecordingsTable without having to replicate the same information across all repetitions and sequential-, simultaneous-, and individual intracellular recordings. For analysis, this means that we can easily focus on individual aspects of an experiment while still being able to easily access information about information from related tables. All of these tables are optional, but to use one you must use all of the lower level tables, even if you only need a single row.
Add a simultaneous recording
The SimultaneousRecordingsTable groups intracellular recordings from the IntracellularRecordingsTable together that were recorded simultaneously from different electrodes and/or cells and describes metadata that is constant across the simultaneous recordings. In practice a simultaneous recording is often also referred to as a sweep. This example adds a custom column, "simultaneous_recording_tag."
% create simultaneous recordings table with custom column
'description', 'A custom tag for simultaneous_recordings', ...
'data', {'LabTag1'} ...
) ...
);
Depending on the lab workflow, it may be useful to add complete columns to a table after we have already populated the table with rows. That would be done like so:
The SequentialRecordingsTable groups simultaneously recorded intracellular recordings from the SimultaneousRecordingsTable together and describes metadata that is constant across the simultaneous recordings. In practice a sequential recording is often also referred to as a sweep sequence. A common use of sequential recordings is to group together simultaneous recordings where a sequence of stimuli of the same type with varying parameters have been presented in a sequence (e.g., a sequence of square waveforms with varying amplitude).
The RepetitionsTable groups sequential recordings from the SequentialRecordingsTable. In practice, a repetition is often also referred to a run. A typical use of the RepetitionsTable is to group sets of different stimuli that are applied in sequence that may be repeated.
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/images.html b/docs/source/_static/html/tutorials/images.html
new file mode 100644
index 00000000..bbdaecdf
--- /dev/null
+++ b/docs/source/_static/html/tutorials/images.html
@@ -0,0 +1,371 @@
+
+Storing Image Data in NWB
Storing Image Data in NWB
Image data can be a collection of individual images or movie segments (as a movie is simply a series of images), about the subject, the environment, the presented stimuli, or other parts related to the experiment. This tutorial focuses in particular on the usage of:
OpticalSeries: Storing series of images as stimuli
OpticalSeriesis for time series of images that were presented to the subject as stimuli. We will create anOpticalSeriesobject with the name"StimulusPresentation"representing what images were shown to the subject and at what times.
Image data can be stored either in the HDF5 file or as an external image file. For this tutorial, we will use fake image data with shape of('time', 'x', 'y', 'RGB')=(200,50,50,3). As in allTimeSeries, the first dimension is time. The second and third dimensions represent x and y. The fourth dimension represents the RGB value (length of 3) for color images. Please note: As described in the dimensionMapNoDataPipes tutorial, when a MATLAB array is exported to HDF5, the array is transposed. Therefore, in order to correctly export the data, we will need to create a transposed array, where the dimensions are in reverse order compared to the type specification.
NWB differentiates between acquired data and data that was presented as stimulus. We can add it to theNWBFileobject as stimulus data.
If the sampling rate is constant, use rate and starting_time to specify time. For irregularly sampled recordings, use timestamps to specify time for each sample image.
AbstractFeatureSeries: Storing features of visual stimuli
While it is usually recommended to store the entire image data as anOpticalSeries, sometimes it is useful to store features of the visual stimuli instead of or in addition to the raw image data. For example, you may want to store the mean luminance of the image, the contrast, or the spatial frequency. This can be done using an instance ofAbstractFeatureSeries. This class is a general container for storing time series of features that are derived from the raw image data.
% Create some fake feature data
feature_data = rand(3, 200); % 200 time points, 3 features
ImageSeries: Storing series of images as acquisition
ImageSeriesis a general container for time series of images acquired during the experiment. Image data can be stored either in the HDF5 file or as an external image file. When color images are stored in the HDF5 file the color channel order is expected to be RGB.
image_data = randi(255, [3, 50, 50, 200]);
behavior_images = types.core.ImageSeries( ...
'data', image_data, ...
'description', 'Image data of an animal in environment', ...
External files (e.g. video files of the behaving animal) can be added to theNWBFileby creating anImageSeriesobject using theexternal_fileattribute that specifies the path to the external file(s) on disk. The file(s) path must be relative to the path of the NWB file. Eitherexternal_fileordatamust be specified, but not both. external_file can be a cell array of multiple video files.
The starting_frame attribute serves as an index to indicate the starting frame of each external file, allowing you to skip the beginning of videos.
Static images can be stored in anNWBFileobject by creating anRGBAImage,RGBImageorGrayscaleImageobject with the image data. All of these image types provide an optional description parameter to include text description about the image and the resolution parameter to specify the pixels/cm resolution of the image.
RGBAImage: for color images with transparency
RGBAImage is for storing data of color image with transparency. data must be 3D where the first and second dimensions represent x and y. The third dimension has length 4 and represents the RGBA value.
image_data = randi(255, [4, 200, 200]);
rgba_image = types.core.RGBAImage( ...
'data', image_data, ... % required
'resolution', 70.0, ...
'description', 'RGBA image' ...
);
RGBImage: for color images
RGBImageis for storing data of RGB color image.datamust be 3D where the first and second dimensions represent x and y. The third dimension has length 3 and represents the RGB value.
image_data = randi(255, [3, 200, 200]);
rgb_image = types.core.RGBImage( ...
'data', image_data, ... % required
'resolution', 70.0, ...
'description', 'RGB image' ...
);
GrayscaleImage: for grayscale images
GrayscaleImageis for storing grayscale image data.datamust be 2D where the first and second dimensions represent x and y.
image_data = randi(255, [200, 200]);
grayscale_image = types.core.GrayscaleImage( ...
'data', image_data, ... % required
'resolution', 70.0, ...
'description', 'Grayscale image' ...
);
Images: a container for images
Add the images to anImagescontainer that accepts any of these image types.
image_collection = types.core.Images( ...
'description', 'A collection of logo images presented to the subject.'...
You may want to set up a time series of images where some images are repeated many times. You could create an ImageSeries that repeats the data each time the image is shown, but that would be inefficient, because it would store the same data multiple times. A better solution would be to store the unique images once and reference those images. This is how IndexSeriesworks. First, create an Images container with the order of images defined using an ImageReferences. Then create an IndexSeries that indexes into the Images.
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/intro.html b/docs/source/_static/html/tutorials/intro.html
new file mode 100644
index 00000000..63ebd0ce
--- /dev/null
+++ b/docs/source/_static/html/tutorials/intro.html
@@ -0,0 +1,307 @@
+
+Introduction to MatNWB
An NWB file represents a single session of an experiment. Each file must have a session_description, identifier, and session start time. Create a new NWBFile object with those and additional metadata using the NwbFile command. For all MatNWB classes and functions, we use the Matlab method of entering keyword argument pairs, where arguments are entered as name followed by value. Ellipses are used for clarity.
nwb = NwbFile( ...
'session_description', 'mouse in open exploration',...
You can also provide information about your subject in the NWB file. Create a Subject object to store information such as age, species, genotype, sex, and a freeform description. Then set nwb.general_subject to the Subject object.
Each of these fields is free-form, so any values will be valid, but here are our recommendations:
For species, we recommend using the formal latin binomal name (e.g. mouse -> Mus musculus, human -> Homo sapiens)
For sex, we recommend using F (female), M (male), U (unknown), and O (other)
subject = types.core.Subject( ...
'subject_id', '001', ...
'age', 'P90D', ...
'description', 'mouse 5', ...
'species', 'Mus musculus', ...
'sex', 'M' ...
);
nwb.general_subject = subject;
subject
Note: the DANDI archive requires all NWB files to have a subject object with subject_id specified, and strongly encourages specifying the other fields.
Time Series Data
TimeSeriesis a common base class for measurements sampled over time, and provides fields fordataandtimestamps(regularly or irregularly sampled). You will also need to supply thenameandunitof measurement (SI unit).
For instance, we can store a TimeSeriesdata where recording started0.0seconds afterstart_timeand sampled every second (1 Hz):
The TimeSeries class serves as the foundation for all other time series types in the NWB format. Several specialized subclasses extend the functionality of TimeSeries, each tailored to handle specific kinds of data. In the next section, we’ll explore one of these specialized types. For a full overview, please check out the type hierarchy in the NWB schema documentation.
Other Types of Time Series
As mentioned previously, there are many subtypes of TimeSeries in MatNWB that are used to store different kinds of data. One example is AnnotationSeries, a subclass of TimeSeries that stores text-based records about the experiment. Similar to our TimeSeries example above, we can create an AnnotationSeries object with text information about a stimulus and add it to the stimulus_presentation group in the NWBFile. Below is an example where we create an AnnotationSeries object with annotations for airpuff stimuli and add it to the NWBFile.
% Create an AnnotationSeries object with annotations for airpuff stimuli
annotations = types.core.AnnotationSeries( ...
'description', 'Airpuff events delivered to the animal', ...
Many types of data have special data types in NWB. To store the spatial position of a subject, we will use the SpatialSeries and Position classes.
Note: These diagrams follow a standard convention called "UML class diagram" to express the object-oriented relationships between NWB classes. For our purposes, all you need to know is that an open triangle means "extends" and an open diamond means "is contained within." Learn more about class diagrams on the wikipedia page.
SpatialSeries is a subclass of TimeSeries, a common base class for measurements sampled over time, and provides fields for data and time (regularly or irregularly sampled). Here, we put a SpatialSeries object called'SpatialSeries' in aPosition object. If the data is sampled at a regular interval, it is recommended to specify the starting_time and the sampling rate (starting_time_rate), although it is still possible to specify timestamps as in the time_series_with_timestamps example above.
NWB differentiates between raw, acquired data, which should never change, and processed data, which are the results of preprocessing algorithms and could change. Let's assume that the animal's position was computed from a video tracking algorithm, so it would be classified as processed data. Since processed data can be very diverse, NWB allows us to create processing modules, which are like folders, to store related processed data or data that comes from a single algorithm.
Create a processing module called "behavior" for storing behavioral data in the NWBFile and add the Position object to the module.
% add the processing module to the NWBFile object, and name the processing module "behavior"
nwb.processing.set('behavior', behavior_mod);
Trials
Trials are stored in a TimeIntervals object which is a subclass of DynamicTable. DynamicTable objects are used to store tabular metadata throughout NWB, including for trials, electrodes, and sorted units. They offer flexibility for tabular data by allowing required columns, optional columns, and custom columns.
The trials DynamicTable can be thought of as a table with this structure:
Trials are stored in a TimeIntervals object which subclasses DynamicTable. Here, we are adding'correct', which will be a logical array.
We can print the SpatialSeries data traversing the hierarchy of objects. The processing module called 'behavior' contains our Position object named 'Position'. The Position object contains our SpatialSeries object named 'SpatialSeries'.
Counter to normal MATLAB workflow, data arrays are read passively from the file. Calling read_spatial_series.data does not read the data values, but presents a DataStub object that can be indexed to read data.
read_spatial_series.data
This allows you to conveniently work with datasets that are too large to fit in RAM all at once. Access all the data in the matrix using the load method with no arguments.
read_spatial_series.data.load
If you only need a section of the data, you can read only that section by indexing the DataStub object like a normal array in MATLAB. This will just read the selected region from disk into RAM. This technique is particularly useful if you are dealing with a large dataset that is too big to fit entirely into your available RAM.
read_spatial_series.data(:, 1:10)
Next Steps
This concludes the introductory tutorial. Please proceed to one of the specialized tutorials, which are designed to follow this one.
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/ogen.html b/docs/source/_static/html/tutorials/ogen.html
new file mode 100644
index 00000000..4e19478b
--- /dev/null
+++ b/docs/source/_static/html/tutorials/ogen.html
@@ -0,0 +1,201 @@
+
+Optogenetics
Optogenetics
This tutorial will demonstrate how to write optogenetics data.
Creating an NWBFile object
When creating a NWB file, the first step is to create theNWBFileobject using NwbFile.
nwb = NwbFile( ...
'session_description', 'mouse in open exploration',...
Theogen module contains two data types that you will need to write optogenetics data,OptogeneticStimulusSite, which contains metadata about the stimulus site, andOptogeneticSeries, which contains the values of the time series.
First, you need to create aDeviceobject linked to theNWBFile:
With theOptogeneticStimulusSiteadded, you can now create and add aOptogeneticSeries. Here, we will generate some random data and specify the timing usingrate. If you have samples at irregular intervals, you should usetimestampsinstead.
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/ophys.html b/docs/source/_static/html/tutorials/ophys.html
new file mode 100644
index 00000000..c9cab142
--- /dev/null
+++ b/docs/source/_static/html/tutorials/ophys.html
@@ -0,0 +1,479 @@
+
+MatNWB Optical Physiology Tutorial
In this tutorial, we will create fake data for a hypothetical optical physiology experiment with a freely moving animal. The types of data we will convert are:
Acquired two-photon images
Image segmentation (ROIs)
Fluorescence and dF/F response
It is recommended to first work through the Introduction to MatNWB tutorial, which demonstrates installing MatNWB and creating an NWB file with subject information, animal position, and trials, as well as writing and reading NWB files in MATLAB.
Set up the NWB file
An NWB file represents a single session of an experiment. Each file must have a session_description, identifier, and session start time. Create a new NWBFile object with those and additional metadata. For all MatNWB functions, we use the Matlab method of entering keyword argument pairs, where arguments are entered as name followed by value.
nwb = NwbFile( ...
'session_description', 'mouse in open exploration',...
Optical physiology results are written in four steps:
Create imaging plane
Acquired two-photon images
Image segmentation
Fluorescence and dF/F responses
Imaging Plane
First, you must create an ImagingPlane object, which will hold information about the area and method used to collect the optical imaging data. This requires creation of a Device object for the microscope and an OpticalChannel object. Then you can create an ImagingPlane.
optical_channel = types.core.OpticalChannel( ...
'description', 'description', ...
'emission_lambda', 500.);
device = types.core.Device();
nwb.general_devices.set('Device', device);
imaging_plane_name = 'imaging_plane';
imaging_plane = types.core.ImagingPlane( ...
'optical_channel', optical_channel, ...
'description', 'a very interesting part of the brain', ...
ROIs can be added to a PlaneSegmentation either as an image_mask or as a pixel_mask. An image mask is an array that is the same size as a single frame of the TwoPhotonSeries, and indicates where a single region of interest is. This image mask may be boolean or continuous between 0 and 1. A pixel_mask, on the other hand, is a list of indices (i.e coordinates) and weights for the ROI. The pixel_mask is represented as a compound data type using a ragged array and below is an example demonstrating how to create either an image_mask or a pixel_mask. Changing the dropdown selection will update the PlaneSegmentation object accordingly.
Now that ROIs are stored, you can store fluorescence dF/F data for these regions of interest. This type of data is stored using the RoiResponseSeries class. You will not need to instantiate this class directly to create objects of this type, but it is worth noting that this is the class you will work with after you read data back in.
First, create a data interface to store this data in
if isfile(nwb_file_name); delete(nwb_file_name); end
nwbExport(nwb, nwb_file_name);
Reading the NWB file
read_nwb = nwbRead(nwb_file_name, 'ignorecache');
Data arrays are read passively from the file. CallingTimeSeries.data does not read the data values, but presents an HDF5 object that can be indexed to read data.
NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+
If all you need is a section of the data, you can read only that section by indexing the DataStub object like a normal array in MATLAB. This will just read the selected region from disk into RAM. This technique is particularly useful if you are dealing with a large dataset that is too big to fit entirely into your available RAM.
read_nwb.processing.get('ophys'). ...
nwbdatainterface.get('Fluorescence'). ...
roiresponseseries.get('RoiResponseSeries'). ...
data(1:5, 1:10)
ans = 5×10
NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+ NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
+
% read back the image/pixel masks and display the first roi
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/ophys_tutorial_schematic.png b/docs/source/_static/html/tutorials/ophys_tutorial_schematic.png
new file mode 100644
index 00000000..7e8a94e4
Binary files /dev/null and b/docs/source/_static/html/tutorials/ophys_tutorial_schematic.png differ
diff --git a/docs/source/_static/html/tutorials/read_demo.html b/docs/source/_static/html/tutorials/read_demo.html
new file mode 100644
index 00000000..46c9dafb
--- /dev/null
+++ b/docs/source/_static/html/tutorials/read_demo.html
@@ -0,0 +1,356 @@
+
+Reading NWB Data in MATLAB
Reading NWB Data in MATLAB
Authors: Ryan Ly, with modification by Lawrence Niu
Last Updated: 2023-09-05
Introduction
In this tutorial, we will read single neuron spiking data that is in the NWB standard format and do a basic visualization of the data. More thorough documentation regarding reading files as well as the NwbFile class, can be found in the NWB Overview Documentation
A NWB file represents a single session of an experiment. It contains all the data of that session and the metadata required to understand the data.
We will use data from one session of an experiment byChandravadia et al. (2020), where the authors recorded single neuron electrophysiological activity from the medial temporal lobes of human subjects while they performed a visual recognition memory task.
Toward the top middle of the page, click the "Files" button.
3. Click on the folder "sub-P11MHM" (click the folder name, not the checkbox).
4. Then click on the download symbol to the right of the filename "sub-P11HMH_ses-20061101_ecephys+image.nwb" to download the data file (69 MB) to your computer.
Installing matnwb
Use the code below to install MatNWB from source using git. Ensure git is on your path before running this line.
MatNWB works by automatically creating API classes based on the schema. For most NWB files, the classes are generated automatically by calling nwbRead farther down. This particular NWB file was created before this feature was supported, so we must ensure that these classes for the correct schema versions are properly generated before attempting to read from the file.
% add the path to matnwb and generate the core classes
addpath('matnwb');
% Reminder: YOU DO NOT NORMALLY NEED TO CALL THIS FUNCTION. Only attempt this method if you
You can also use util.nwbTree to actively explore the NWB file.
util.nwbTree(nwb);
Stimulus
Now lets take a look at the visual stimuli presented to the subject. They will be in nwb.stimulus_presentation
nwb.stimulus_presentation
ans =
Set with properties:
+
+ StimulusPresentation: [types.core.OpticalSeries]
+
This results shows us that nwb.stimulus_presentation is a Set object that contains a single data object called StimulusPresentation, which is an OpticalSeries neurodata type. Use the get method to return this OpticalSeries. Set objects store a collection of other NWB objects.
OpticalSeries is a neurodata type that stores information about visual stimuli presented to subjects. This print out shows all of the attributes in the OpticalSeries object named StimulusPresentation. The images are stored in StimulusPresentation.data
When calling a data object directly, the data is not read but instead a DataStub is returned. This is because data is read "lazily" in MatNWB. Instead of reading the entire dataset into memory, this provides a "window" into the data stored on disk that allows you to read only a section of the data. In this case, the last dimension indexes over images. You can index into any DataStub as you would any MATLAB matrix.
% get the image and display it
% the dimension order is provided as follows:
% [rgb, y, x, image index]
img = StimulusImageData(1:3, 1:300, 1:400, 32);
A bit of manipulation allows us to display the image using MATLAB's imshow.
img = permute(img,[3, 2, 1]); % fix orientation
img = flip(img, 3); % reverse color order
F = figure();
imshow(img, 'InitialMagnification', 'fit');
daspect([3, 5, 5]);
To read an entire dataset, use the DataStub.load method without any input arguments. We will use this approach to read all of the image display timestamps into memory.
Spike times from this unit are aligned to each stimulus time and compiled in a cell array
results = cell(1, length(stimulus_times));
for itime = 1:length(stimulus_times)
stimulus_time = stimulus_times(itime);
spikes = unit_spikes - stimulus_time;
spikes = spikes(spikes > -before);
spikes = spikes(spikes < after);
results{itime} = spikes;
end
Plot results
Finally, here is a (slightly sloppy) peri-stimulus time histogram
figure();
hold on
for i = 1:length(results)
spikes = results{i};
yy = ones(length(spikes)) * i;
plot(spikes, yy, 'k.');
end
hold off
ylabel('trial');
xlabel('time (s)');
axis('tight')
figure();
all_spikes = cat(1, results{:});
histogram(all_spikes, 30);
ylabel('count')
xlabel('time (s)');
axis('tight')
Conclusion
This is an example of how to get started with understanding and analyzing public NWB datasets. This particular dataset was published with an extensive open analysis conducted in both MATLAB and Python, which you can find here. For more datasets, or to publish your own NWB data for free, check out the DANDI archive here. Also, make sure to check out the DANDI breakout session later in this event.
+
+
+
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/remote_read.html b/docs/source/_static/html/tutorials/remote_read.html
new file mode 100644
index 00000000..9746120e
--- /dev/null
+++ b/docs/source/_static/html/tutorials/remote_read.html
@@ -0,0 +1,58 @@
+
+Remote read of NWB files
Remote read of NWB files
It is possible to read an NWB file (or any HDF5 file) in MATLAB directly from several different kinds of remote locations, including AWS, Azure Blob Storage and HDFS. This tutorial will walk you through specifically loading a MATLAB file from AWS S3, which is the storage used by the DANDI archive. See MATLAB documentation for more general information.
To read an NWB file file from an s3 store, first you need to figure out the s3 path of that resource. The easiest way to do this is to use the DANDI web client.
(skip if on DANDI Hub) Make sure you do not have a file ~/.aws/credentials. If you do, rename it to something else. On Windows this file would be somewhere like C:/Users/username/.aws/credentials.
Find and select a dandiset you want on the DANDI Archive, then click
Navigate to the NWB file of interest and click
Find the second entry of "contentURL"
In your MATLAB session, take the end of that url (the blob id) and add it to this expression: s3 = 's3://dandiarchive/blobs/<blob_id>'. In this case, you would have:
That's it! MATLAB will automatically detect that this is an S3 path instead of a local filepath and will set up a remote read interface for that NWB file. This appoach works on any computer with a fairly recent version of MATLAB and an internet connection. It works particularly well on the DANDI Hub, which has a very fast connection to the DANDI S3 store and which provides a MATLAB environment for free provided you have a license.
Note: MATLAB vs. Python remote read
Python also allows you to remotely read a file, and has several advantages over MATLAB. Reading in Python is faster. On DANDI Hub, for MATLAB, reading the file takes about 51 seconds, while the analogous operation takes less than a second in Python. Python also allows you to create a local cache so you are not repeatedly requesting the same data, which can further speed up data access. Overall, we recommend remote reading using Python instead of MATLAB.
+
+
+
\ No newline at end of file
diff --git a/docs/source/_static/html/tutorials/scratch.html b/docs/source/_static/html/tutorials/scratch.html
new file mode 100644
index 00000000..fe467263
--- /dev/null
+++ b/docs/source/_static/html/tutorials/scratch.html
@@ -0,0 +1,164 @@
+
+Scratch Data
Scratch Data
This tutorial will focus on the basics of working with a NWBFile for storing non-standardizable data. For example, you may want to store results from one-off analyses of some temporary utility. NWB provides in-file scratch space as a dedicated location where miscellaneous non-standard data may be written.
'description', 'a module to store filtering results', ...
'filtered_timeseries', FilteredTs ...
);
ContextFile.processing.set('core', ProcModule);
nwbExport(ContextFile, 'context_file.nwb');
Warning Regarding the Usage of Scratch Space
Scratch data written into the scratch space should not be intended for reuse or sharing. Standard NWB types, along with any extensions, should always be used for any data intended to be shared. Published data should not include scratch data and any reuse should not require scratch data for data processing.
Writing Data to Scratch Space
Let us first copy what we need from the processed data file.