Bristlemouth Mote Filesystem

Community
1

Hello Bristlemouth community!
I am embarking on a journey to add a filesystem to the Bristlemouth mote for use in Dev Kit applications as well as new product initiatives!

Why does this matter?
Well, imagine a future where Bristlemouth devices are able to transfer files to one another over an FTP like service.
This would unlock the door for so many interesting applications:

  • Collecting coredumps from nodes
  • Sending configuration files directly to a node rather than individual parameters
  • Passing along audio data from a hydrophone to a high bandwidth telemetry modem

The possibilities are endless1

How could the mote support this?
The mote has an 8MB flash chip available for use.
Currently only about 30kB of it is being used for the mote configuration partitions :duck:.
This means there is about 99.6% of the flash chip is available!
Sounds like there is enough space to throw a file system partition on the external flash chip.
A point to mention, the external flash is not even used to house DFU images (if it was we could have another 1MB of internal flash to use on the mote’s Microcontroller :laughing:).

So what filesystem should I implement you ask?
Well, there are a few filesystems that exist out there that are open source and geared towards embedded system (small computer) applications:

  • uC-FS
    • Apart of the uC-OS real time operating system by Micrium, looks cool, but looks very specific to their operating system
  • Little-FS
    • Could almost be considered industry standard, something I have personally used in the past and have had a pretty solid experience with
  • spiffs
    • Widely used, is the main file system on the ESP32’s ecosystem (super popular hobby Microcontroller), has not been maintained in the last 8 years

All of these are great options and with enough effort (some more than others) could be integrated into bm_protocol.
But let me introduce you to another filesystem: STORfs.
This is a filesystem that I wrote when I first began my career about 5 years ago.
It is messy.
It is inefficient.
It is ugly.
But, it is mine and open source (and for some reason gaining a lot of traction on Github, it is up to 37 starts at the time of this post).
I want to make it better and hopefully make it a worthwhile filesystem for all your ocean needs!

In order to get there, there are a few things that need to happen in STORfs:

  • Restructure of files/organization, a major refactor of the code
  • Improved searching for files
    • Currently a tree-structure is used for directories and a linked list for files in the directory, I plan to switch to hashtables with an inode like structure
  • Journaling and improved caching
  • Tests
    • Performance tests
    • Unit tests

I will be keeping this thread updated with my journey along the way, I hope something like this excites you all as much as it does me :slight_smile:

4 Likes
2

Awesome! After writing my last post, I’ve spoken this work into the world, which means I’m now committed to actually following through. Since then, I’ve already made some initial strides.

First, let me lay out a roadmap for how I plan to accomplish the improvements to STORfs.

What are those improvements again?

The most significant improvement will be how directories and files are searched for and organized within the filesystem. This new approach will improve both the speed of file access and the efficiency of flash block utilization.

Here are the steps to make this a production-ready library:

  • Perform an extraction refactor of the codebase. This encapsulates major operations into their own files, making the code more maintainable and digestible while enabling easy unit testing of modularized components.

  • Add unit testing infrastructure. I’ve used Ceedling in past projects and really enjoyed its flexibility and ease of setup. Check out how I used it in my polyglot project on GitHub.

  • Implement the new file access scheme. This will include:

    1. Using a hash table for organizing directory contents
    2. Using an inode-like structure (which I’ll rename to snode for STORfs) to store file metadata

  • Add atomic operations to prevent filesystem corruption in the event of power loss. This could range from a simple write-sync-read-validate operation to a full journaling implementation.

  • Integrate the new file access scheme and atomic operations into the existing codebase. This will deliver the improvements needed for faster filesystem operations.

  • Test on actual hardware:

    1. x86/ARM Mac
    2. A Bristlemouth mote
    3. Other PCBs I’ve designed for personal projects, to gather additional data points

    Once implemented on these platforms, the goal is to benchmark performance. I’ve looked for de-facto filesystem benchmark tools but haven’t found a standard yet—I’ll do more research when the time comes.

  • Proper wear leveling implementation. The current approach isn’t true wear leveling and has some flaws (which I’ll detail in a future blog post).

  • Better hooks for thread safety. Combined with caching of open files, this will allow files to be accessed from multiple threads quickly and safely.

Bonus features (not required, but nice to have):

  • Implement garbage collection to help with defragmentation and reclaim space from deleted files.
  • Filesystem check and repair utilities in case of filesystem corruption.

Progress Update:

At the time of writing, the modularization (first step) has been completed, and I’ve begun work on the unit test framework.

For each of these steps, I plan to write another post diving into more detail about the work done!

2 Likes
3

In this post, I’m going to discuss how I’m tracking the work being done, as well as the extraction refactor from the aforementioned first phase.

Within the STORfs repo I have created some issues that will be used to track the work described in my previous post. Issues on Github are not necessarily problems (bugs) in the repository, but can be used to capture a general unit of work or pertinent discussion as well.These issues are mainly being used to keep me on track as well as capture any thoughts that might come up during the implementation of said issues.

When an issue is being worked on, a pull request will be created to associate the issue with the actual code being written. These pull requests will be merged into the develop branch with the full implementation for the phase.

Onto the first order of business: the extraction refactor. This extraction refactor involves identifying and removing cohesive sections of code from a large complex file, to smaller more cohesive files. This makes the code more digestible and maintainable. I like to think of it as breaking down the file into smaller units (this becomes very important when conceptualizing the second phase : Add unit testing infrastructure). In an ideal world, this refactor can all happen without having to structurally change any of the code (i.e., change the logic)—essentially by copy-and-pasting directly to the new files!

I decided to break down the single source file storfs.c into the following file structure:

src/
├─ core.c
├─ crc.c
├─ fgets.c
├─ fopen.c
├─ fputs.c
├─ mkdir.c
├─ mount.c
├─ rewind.c
├─ rm.c
├─ touch.c
├─ wear.c

The refactor breaks down the original file, storfs.c as so:

  • Core Elements
    • Helper functions
    • Root directory operations
  • CRC Calculating Logic
  • Wear Leveling
  • Filesystem operations
    • fopen
    • fgets
    • fputs
    • mount
    • mkdir
    • rewind
    • rm
    • touch

When performing the refactor I would constantly run the test program located at Examples/test in order to make sure the output was consistent with every function moved from storfs.c to one of the newly created files.
This ensured that the changes were not breaking.

The work done can be seen at the following pull request #4. Check it out for a more detailed look at the file structure and layout.

2 Likes