Every BMI research group that has built a recording and stimulation system from hardware up has also built a data format. That format started as a convenience: a struct that matched the firmware's internal representation, serialized to binary and written to a file. Over time, it accumulated fields for new data types, versioning flags, and session metadata. After three years of use, it is the de facto standard for everyone in that lab, and it is incompatible with everything outside it.
This is not a failure of engineering judgment. The format was the right choice when it was built, because the lab's tooling was designed around it and changing it would have broken a year of analysis scripts. The problem only becomes visible when the lab tries to collaborate with a group using different hardware, or when a device maker tries to build on top of the lab's results and has to write a custom importer before any work can begin.
The fragmentation of neural data formats is one of the most concrete friction points in closed-loop BMI development. This post looks at where existing standards have gotten traction, where the gaps remain, and what a common transport layer for closed-loop middleware could look like.
What NWB Got Right
Neurodata Without Borders (NWB) is the closest thing the electrophysiology community has to a shared data format standard. It was developed through a collaboration across major neuroscience institutions and is now the required format for data submissions to the Allen Institute and several other data repositories. NWB covers: extracellular electrophysiology, intracellular recordings, optical physiology, behavioral data, and the metadata needed to make all of those interpretable without out-of-band documentation.
The NWB format is built on HDF5, which gives it good tooling support (h5py, MATLAB HDF5 libraries, and direct C bindings) and efficient storage for large, heterogeneous datasets. The schema is hierarchical and extensible: core types cover the common data modalities, and extensions allow domain-specific data to be added without breaking compatibility with tools that read only the core schema.
For offline analysis of electrophysiology data, NWB is a reasonable choice. The adoption barriers are real (the format is complex and the learning curve is non-trivial for a first-time user), but the ecosystem of analysis tools that speak NWB natively is large enough that the upfront investment pays off for any group that publishes or shares data externally.
Where NWB has not achieved traction is in real-time and embedded contexts. The HDF5 library is not embedded-friendly. Writing an HDF5 file from a Cortex-M4 RTOS is technically possible but carries a large flash and RAM footprint that conflicts with the memory budgets of most neural recording embedded systems. There is no lightweight NWB profile designed for real-time acquisition that then feeds into full NWB archives on a host system.
Where BrainFlow Sits
BrainFlow is a different kind of project. It is not a data format standard; it is a hardware abstraction library that provides a common API for reading data from dozens of commercial and research EEG and BCI headsets. The output of a BrainFlow acquisition session is a simple matrix: channels by samples, with a timestamp row, no schema, no metadata beyond what the hardware adapter provides.
BrainFlow's strength is the breadth of supported hardware. If you want to read data from an Emotiv headset, an OpenBCI Cyton, a Muse 2, and a Neurosity Crown using the same 10 lines of Python, BrainFlow is the tool. The library handles the device-specific connection protocol, the packet framing, and the signal preprocessing steps specific to each device.
The limitation is that BrainFlow's data model is too thin to carry the information needed for closed-loop middleware. Stimulation events, decoder outputs, safety envelope configurations, and session state transitions have no representation in a channels-by-samples matrix. BrainFlow is the right tool for the acquisition end of a recording pipeline; it is not a transport layer for closed-loop system data.
The Gap: No Standard Transport for Closed-Loop Events
What does not exist is a standard format or protocol for the event stream that a closed-loop BMI system generates: the sequence of detected spike events, decoded feature outputs, stimulation commands dispatched, parameter adaptation events, safety-limit violations, and session state transitions that together describe what happened during a closed-loop session.
Every closed-loop system we have encountered logs this event stream in a proprietary format. Some use structured binary records. Some use timestamped CSV files. Some use UDP datagrams with a custom message schema. None of these are directly compatible with any other system's tooling.
This matters because it creates friction at every collaboration boundary. When a device maker integrates a closed-loop system with a research lab's analysis pipeline, someone has to write a parser. When a device validation team needs to audit a session log against a protocol specification, the audit tooling has to be custom-built. When two research groups want to compare the behavior of their respective closed-loop systems on the same task, the comparison requires a data harmonization step before any analysis begins.
What a Common Transport Layer Would Look Like
The minimal viable common transport layer for closed-loop middleware would need to cover three things: a typed event record schema, a timestamp format that can be synchronized across devices, and a session container that links event records to the protocol configuration that governed the session.
The event record schema needs to cover the basic closed-loop event types: spike detection events (channel, timestamp, feature vector), decoded state events (timestamp, feature class, confidence if available), stimulation command events (channel, timestamp, pulse parameters), stimulation anomaly events (command rejected and the reason), and session state transitions (timestamp, previous state, new state). These six event types cover the majority of what a closed-loop session log needs to contain.
The timestamp format needs to be precise enough to reconstruct the timing relationships between recording and stimulation events. Millisecond resolution is insufficient for closed-loop systems where the control loop operates at kilohertz rates. Microsecond timestamps referenced to a session-start anchor are the minimum, with a mechanism for synchronizing the session anchor across multiple devices when the recording hardware and the stimulation hardware are separate physical units.
The session container is the metadata layer: which protocol configuration was loaded, which electrode configuration was in use, which firmware version generated the log, and when the session started and ended. This is the information that allows an audit tool to verify that the session was conducted within the bounds of an approved protocol.
Ruten's Approach to This Problem
In the Ruten SDK, we have implemented a session log format that covers all six event types described above, using a compact binary schema with a reference Python reader. The format is not yet a proposed standard for the field; it is a format that works for our middleware's use case and that we have designed to be compatible with NWB at the session level (the session container metadata maps to NWB session-level metadata fields).
We have also made the format specification public. The wire format, the type codes, and the field layouts are documented in the SDK, not treated as internal implementation details. That decision reflects a practical judgment: a format that is proprietary to Ruten has limited value to the broader device development community. A format that is well-documented and compatible with existing NWB tooling for the parts of the schema that overlap is at least a candidate for adoption by teams that want a starting point rather than building from scratch.
We are not claiming that the Ruten session log format should be the standard. We are saying that the field would benefit from any convergence toward a shared approach, and that convergence is more likely to happen through open documentation and feedback from users than through a top-down standardization process. The open standards working group participation mentioned in our research partnership program is where we are putting effort into that conversation.
The Practical Value of Convergence
The case for a common transport standard is not primarily about elegance. It is about reducing the cost of the tasks that every closed-loop BMI development team has to do: log analysis, protocol compliance verification, cross-system comparison. Each of those tasks takes longer and requires more custom tooling when every system speaks a different format.
A format converges when the cost of adopting it is lower than the cost of maintaining a custom alternative. For NWB in offline electrophysiology, that convergence happened when the data sharing requirements from major funders made NWB adoption nearly mandatory. For real-time closed-loop event streams, the forcing function is less obvious. It may come from device validation requirements that specify what a session log must contain in order to be auditable. It may come from a widely-adopted analysis toolkit that only reads a specific format. It may come from a middleware library that enough device makers use that its session log format becomes a de facto standard by volume.
The work needed to get there is less technically complex than NWB was. A closed-loop event stream schema does not need to cover the full breadth of neuroscience data modalities. It needs to cover a small number of event types consistently and precisely. That is a tractable problem, and solving it would reduce friction at one of the most common integration points in neuro-rehabilitation device development.