Application Development for Integrating YSI EXO2 Sonde into Smart Mooring using Bristlemouth

My name is Cobi Christiansen and I am the Data Operations Specialist at Coastal Carolina University. I am working on integrating a YSI EXO2 water quality sonde that has the following sensors: temperature, conductivity, dissolved oxygen, pH, and turbidity. The physical integration is nearly complete, but I am still working on developing an application that parses the incoming sonde data, aggregates data over a period of time, calculates min, max, mean, and standard deviation over aggregation period, converts stats to number format, and finally sends data to the bristlemouth network. I am trying to modify the existing code from src/apps/bm_devkit/rbr_coda_example. I have some coding experience, but C++ is new to me.

I am struggling to get the parser to work with my line of data (outlined at the top of code below). I reached out to YSI to see if the sonde can spit out just the data values without the #, but I have not heard back yet.

Any assistance with this code would be greatly appreciated!

/*
 * This is a lightweight example Dev Kit application (v.0.6.0) to integrate a YSI EXO2 Sonde.
 * Sonde Paramters: temp, conductivity, DO, pH
 * Example line: # 21.853 0.790 0.00 7.58 
 * Current Issue 1: Parsing the "#" in the line... "ERR - can't parse value of ValueType 0"
    Solution?: Find the correct way to parse the "#" or find a way to exclude the first character when parsing
 * Current Issue 2: It does not seem to be aggregating the data correctly
    Solution?: Check to make sure exo_data is pulling correctly. Does this have to do with the parsing issue?
 */

#include "user_code.h"
#include "LineParser.h"
#include "OrderedSeparatorLineParser.h"
#include "array_utils.h"
#include "avgSampler.h"
#include "bm_network.h"
#include "bm_printf.h"
#include "bm_pubsub.h"
#include "bristlefin.h"
#include "bsp.h"
#include "debug.h"
#include "lwip/inet.h"
#include "payload_uart.h"
#include "sensors.h"
#include "stm32_rtc.h"
#include "task_priorities.h"
#include "uptime.h"
#include "usart.h"
#include "util.h"

#define LED_ON_TIME_MS 20
#define LED_PERIOD_MS 1000
#define DEFAULT_BAUD_RATE 9600
#define DEFAULT_LINE_TERM 13 // A command is terminated by a <CR>
#define BYTES_CLUSTER_MS 50
// How often to compute and return statistics
#define EXO_AGG_PERIOD_MIN 0.5
// 10 min => 600,000 ms
#define EXO_AGG_PERIOD_MS (EXO_AGG_PERIOD_MIN * 60 * 1000)
// We have enough RAM that we can keep it simple for shorter durations - use 64 bit doubles, buffer all readings.
// We could be much more RAM and precision efficient by using numerical methods like Kahan summation and Welford's algorithm.*/
#define MAX_EXO_SAMPLES ((EXO_AGG_PERIOD_MS / 1000) + 10)
typedef struct {
  uint16_t sample_count;
  double min;
  double max;
  double mean;
  double stdev;
} __attribute__((__packed__)) exoData_t;
#define EXO_DATA_SIZE sizeof(exoData_t)

// Create an instance of the averaging sampler for our data
static AveragingSampler exo_data;

// app_main passes a handle to the user config partition in NVM.
extern cfg::Configuration *userConfigurationPartition;

// A timer variable we can set to trigger a pulse on LED2 when we get payload serial data
static int32_t ledLinePulse = -1;
static u_int32_t baud_rate_config = DEFAULT_BAUD_RATE;
static u_int32_t line_term_config = DEFAULT_LINE_TERM;

// A buffer for our data from the payload uart
char payload_buffer[2048];


/*  For Turning Text Into Numbers...
 *    Setup a LineParser to turn the ASCII serial data from the YSI EXO2 sonde into numbers
 *    Lines from the EXO2 look like: # 21.853 0.790 0.00 7.58 
 *    256 character buffer should be more than enough 
 *    For unsigned ints, let's use 64 bits, and for floating point let's use 64 bit doubles.
 *    We've got luxurious amounts of RAM on this chip, and it's much easier to avoid roll-overs and precision issues
 *    by using it vs. troubleshooting them because we prematurely optimized things.
 *    Declare the parser here with separator, buffer length, value types array, and number of values per line.
 *    We'll initialize the parser later in setup to allocate all the memory we'll need.
 */
ValueType valueTypes[] = {TYPE_INVALID, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE};
OrderedSeparatorLineParser parser(" ", 256, valueTypes, 5);

void setup(void) {
  /* USER ONE-TIME SETUP CODE GOES HERE */

  // Initialize the exo data buffer to be the size we need.
  exo_data.initBuffer(MAX_EXO_SAMPLES);
  // Initialize our LineParser, which will allocated any needed memory for parsing.
  parser.init();
  // Setup the UART – the on-board serial driver that talks to the RS232 transceiver.
  PLUART::init(USER_TASK_PRIORITY);
  // Baud set per expected baud rate of the sensor.
  PLUART::setBaud(baud_rate_config);
  // Enable passing raw bytes to user app.
  PLUART::setUseByteStreamBuffer(true);
  // Enable parsing lines and passing to user app.
  /// Warning: PLUART only stores a single line at a time. If your attached payload sends lines
  /// faster than the app reads them, they will be overwritten and data will be lost.
  PLUART::setUseLineBuffer(true);
  // Set a line termination character per protocol of the sensor.
  PLUART::setTerminationCharacter((char)line_term_config);
  // Turn on the UART.
  PLUART::enable();
  // Enable the input to the Vout power supply.
  bristlefin.enableVbus();
  // ensure Vbus stable before enable Vout with a 5ms delay.
  vTaskDelay(pdMS_TO_TICKS(5));
  // enable Vout, 12V by default.
  bristlefin.enableVout();
}

void loop(void) {
  /* USER LOOP CODE GOES HERE */
  // This aggregates exo readings into stats, and sends them along to Spotter
  static u_int32_t exoStatsTimer = uptimeGetMs();
  if ((u_int32_t)uptimeGetMs() - exoStatsTimer >= EXO_AGG_PERIOD_MS) {
    exoStatsTimer = uptimeGetMs();
    double mean = 0, stdev = 0, min = 0, max = 0;
    uint16_t n_samples = 0;
    if (exo_data.getNumSamples()) {
      mean = exo_data.getMean();
      stdev = exo_data.getStd(mean);
      min = exo_data.getMin();
      max = exo_data.getMax();
      n_samples = exo_data.getNumSamples();
      exo_data.clear();
    }

    // Get the RTC if available
    RTCTimeAndDate_t time_and_date = {};
    rtcGet(&time_and_date);
    char rtcTimeBuffer[32];
    rtcPrint(rtcTimeBuffer, &time_and_date);

    bm_fprintf(0, "payload_data_agg.log",
               "tick: %llu, rtc: %s, n: %u, min: %.4f, max: %.4f, mean: %.4f, "
               "std: %.4f\n",
               uptimeGetMs(), rtcTimeBuffer, n_samples, min, max, mean, stdev);
    bm_printf(0,
              "[exo-agg] | tick: %llu, rtc: %s, n: %u, min: %.4f, max: %.4f, "
              "mean: %.4f, std: %.4f",
              uptimeGetMs(), rtcTimeBuffer, n_samples, min, max, mean, stdev);
    printf("[exo-agg] | tick: %llu, rtc: %s, n: %u, min: %.4f, max: %.4f, "
           "mean: %.4f, std: %.4f\n",
           uptimeGetMs(), rtcTimeBuffer, n_samples, min, max, mean, stdev);
    uint8_t tx_data[EXO_DATA_SIZE] = {};
    exoData_t tx_exo = {
        .sample_count = n_samples, .min = min, .max = max, .mean = mean, .stdev = stdev};
    memcpy(tx_data, (uint8_t *)(&tx_exo), EXO_DATA_SIZE);
    if (spotter_tx_data(tx_data, EXO_DATA_SIZE, BM_NETWORK_TYPE_CELLULAR_IRI_FALLBACK)) {
      printf("%llut - %s | Sucessfully sent Spotter transmit data request\n", uptimeGetMs(),
             rtcTimeBuffer);
    } else {
      printf("%llut - %s | Failed to send Spotter transmit data request\n", uptimeGetMs(),
             rtcTimeBuffer);
    }
  }

  // This checks for a trigger set by ledLinePulse when data is received from the payload UART.
  // Each time this happens, we pulse LED2 Green.
  static bool led2State = false;
  // If LED2 is off and the ledLinePulse flag is set, turn it on Green.
  if (!led2State && ledLinePulse > -1) {
    bristlefin.setLed(2, Bristlefin::LED_GREEN);
    led2State = true;
  }
  // If LED2 has been on for LED_ON_TIME_MS, turn it off.
  else if (led2State && ((u_int32_t)uptimeGetMs() - ledLinePulse >= LED_ON_TIME_MS)) {
    bristlefin.setLed(2, Bristlefin::LED_OFF);
    ledLinePulse = -1;
    led2State = false;
  }

  // This section demonstrates a simple non-blocking bare metal method for rollover-safe timed tasks, like blinking an LED.
  // More canonical (but more arcane) modern methods of implementing this kind functionality would bee to use FreeRTOS tasks or hardware timer ISRs.

  static u_int32_t ledPulseTimer = uptimeGetMs();
  static u_int32_t ledOnTimer = 0;
  static bool led1State = false;

  // Turn LED1 on green every LED_PERIOD_MS milliseconds.
  if (!led1State && ((u_int32_t)uptimeGetMs() - ledPulseTimer >= LED_PERIOD_MS)) {
    bristlefin.setLed(1, Bristlefin::LED_GREEN);
    ledOnTimer = uptimeGetMs();
    ledPulseTimer += LED_PERIOD_MS;
    led1State = true;
  }
  // If LED1 has been on for LED_ON_TIME_MS milliseconds, turn it off.
  else if (led1State && ((u_int32_t)uptimeGetMs() - ledOnTimer >= LED_ON_TIME_MS)) {
    bristlefin.setLed(1, Bristlefin::LED_OFF);
    led1State = false;
  }

  // Read a cluster of bytes if available
  // A timer is used to try to keep clusters of bytes (say from lines) in the same output.
  static int64_t readingBytesTimer = -1;
  // Note - PLUART::setUseByteStreamBuffer must be set true in setup to enable bytes.
  if (readingBytesTimer == -1 && PLUART::byteAvailable()) {
    // Get the RTC if available
    RTCTimeAndDate_t time_and_date = {};
    rtcGet(&time_and_date);
    char rtcTimeBuffer[32];
    rtcPrint(rtcTimeBuffer, &time_and_date);
    printf("[payload-bytes] | tick: %" PRIu64 ", rtc: %s, bytes:", uptimeGetMs(),
           rtcTimeBuffer);
    // not very readable, but it's a compact trick to overload our timer variable with a -1 flag
    readingBytesTimer = (int64_t)((u_int32_t)uptimeGetMs());
  }
  while (PLUART::byteAvailable()) {
    readingBytesTimer = (int64_t)((u_int32_t)uptimeGetMs());
    uint8_t byte_read = PLUART::readByte();
    printf("%02X ", byte_read);
  }
  if (readingBytesTimer > -1 &&
      (u_int32_t)uptimeGetMs() - (u_int32_t)readingBytesTimer >= BYTES_CLUSTER_MS) {
    printf("\n");
    readingBytesTimer = -1;
  }

  // Read a line if it is available
  if (PLUART::lineAvailable()) {
    // Shortcut the raw bytes cluster completion so the parsed line will be on a new console line
    if (readingBytesTimer > -1) {
      printf("\n");
      readingBytesTimer = -1;
    }
    uint16_t read_len = PLUART::readLine(payload_buffer, sizeof(payload_buffer));

    // Get the RTC if available
    RTCTimeAndDate_t time_and_date = {};
    rtcGet(&time_and_date);
    char rtcTimeBuffer[32] = {};
    rtcPrint(rtcTimeBuffer, NULL);
    bm_fprintf(0, "exo_raw.log", "tick: %" PRIu64 ", rtc: %s, line: %.*s\n", uptimeGetMs(),
               rtcTimeBuffer, read_len, payload_buffer);
    bm_printf(0, "[exo] | tick: %" PRIu64 ", rtc: %s, line: %.*s", uptimeGetMs(), rtcTimeBuffer,
              read_len, payload_buffer);
    printf("[exo] | tick: %" PRIu64 ", rtc: %s, line: %.*s\n", uptimeGetMs(), rtcTimeBuffer,
           read_len, payload_buffer);

    // trigger a pulse on LED2
    ledLinePulse = uptimeGetMs();

    // Now when we get a line of text data, our LineParser turns it into numeric values.
    if (parser.parseLine(payload_buffer, read_len)) {
      printf("parsed values: %llu | %f\n", parser.getValue(0).data, parser.getValue(1).data, parser.getValue(2).data), parser.getValue(3).data, parser.getValue(4).data;
    } else {
      printf("Error parsing line!\n");
      return; 
    }

    // Now let's aggregate those values into statistics
    if (exo_data.getNumSamples() >= MAX_EXO_SAMPLES) {
      printf("ERR - No more room in exo reading buffer, already have %d "
             "readings!\n",
             MAX_EXO_SAMPLES);
      return; 
    }

    double exo_reading = parser.getValue(1).data.double_val;
    exo_data.addSample(exo_reading);

    printf("count: %u/%d, min: %f, max: %f\n", exo_data.getNumSamples(),
           MAX_EXO_SAMPLES, exo_data.getMin(), exo_data.getMax());
  }
}
1 Like

Hi Cobi,

Great work! Thanks for sharing. I’m really excited about your integration. A couple small changes are all that’s required. Here’s a diff, and I’ll explain below.

77,78c77,78
< ValueType valueTypes[] = {TYPE_INVALID, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE};
< OrderedSeparatorLineParser parser(" ", 256, valueTypes, 5);
---
> ValueType valueTypes[] = {TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE};
> OrderedSeparatorLineParser parser(" #", 256, valueTypes, 4);
243c243
<       printf("parsed values: %llu | %f\n", parser.getValue(0).data, parser.getValue(1).data, parser.getValue(2).data), parser.getValue(3).data, parser.getValue(4).data;
---
>       printf("parsed values: %f, %f, %f, %f\n", parser.getValue(0).data, parser.getValue(1).data, parser.getValue(2).data, parser.getValue(3).data);

In order to skip the # at the start of the sensor’s output, you can add the # character to the string of delimiters (in addition to space) on line 78.

It’s also important to note that TYPE_INVALID triggers an immediate parsing failure, so you can’t use it in the list of value types. Just pass an array of 4 TYPE_DOUBLE and change the number at the end of line 78 from 5 to 4.

Line 243 which prints ā€œparsed valuesā€ had the closing parenthesis in the wrong place and needed to be modified slightly. Usually the compiler would tell you about that, but the rest of the line just happened to be technically valid C++, so it compiled fine. :man_facepalming:

After these changes the code works as intended, including aggregation. It’s set on line 37 to aggregate and transmit every 30 seconds, which is probably too often. You might burn through a surprising amount of :moneybag: :moneybag: :moneybag: for the telemetry, so I recommend changing EXO_AGG_PERIOD_MIN from 0.5 to 5 as it was in the original example.

You might be surprised by the data that comes from the Sofar API. The example packs the number of samples, min, max, mean, and stdev into a struct for transmission, with each number encoded little endian because that’s how the microcontroller represents the numbers in memory. You need to reverse the order of the bytes to correctly decode them.

Let me explain with an example. I set up another device to behave like the sonde and send your example line # 21.853 0.790 0.00 7.58 every 5 seconds over serial to the dev kit. I see this output from the dev kit console:

[exo-agg] | tick: 5675120, rtc: 2024-07-01T21:06:48.015, n: 5, min: 0.7900, max: 0.7900, mean: 0.7900, std: 0.0000
5675120t - 2024-07-01T21:06:48.015 | Sucessfully sent Spotter transmit data request

And I can see the transmission in the Spotter USB console:

2024-07-01T20:46:48.800Z [MS] [INFO] Added message(id: 386 len: 63) to queue MS_Q_LEGACY: (1)!
Message: DE 66 83 15 B8 DA 01 AC 02 1D A1 2D C6 5D E9 6B 41 71 E5 E2 FC 2D 53 33 39 46 86 2C 20 05 00 48 E1 7A 14 AE 47 E9 3F 48 E1 7A 14 AE 47 E9 3F 48 E1 7A 14 AE 47 E9 3F 00 00 00 00 00 00 00 00 

The first 29 bytes are a header added by the Spotter. After that come your data members. I’ll break them apart with an explanation.

  • 05 00 => 0005: number of readings in the aggregated sample as an unsigned 16-bit integer
  • 48 E1 7A 14 AE 47 E9 3F => 0x3fe947ae147ae148: the low-level double-precision floating-point representation of 0.79. You can verify this and play around using Float Toy. This represents the minimum from the sample.
  • 48 E1 7A 14 AE 47 E9 3F => 0x3fe947ae147ae148 again for the max
  • 48 E1 7A 14 AE 47 E9 3F => 0x3fe947ae147ae148 again for the mean
  • 00 00 00 00 00 00 00 00 => 0: the standard deviation

When you pull this data from the raw-messages API as described in Guide 5, you’ll want to write your own post-processing script to decode the data.

Hope that helps!

1 Like

Thank you for your assistance and explanation. I was still getting a parsing error when I added the # as a separator. I looked closely at what OrderedSeparatorLineParser.cpp wanted and I saw that you can include a header. I changed the code below and it seems to be working as intended now. I am working on cleaning up the code and aggregating the stats from each sensor instead of just one.

77,78c77,78cc77,78
< ValueType valueTypes[] = {TYPE_INVALID, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE};
< OrderedSeparatorLineParser parser(" ", 256, valueTypes, 5);
---
> ValueType valueTypes[] = {TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE};
> OrderedSeparatorLineParser parser(" #", 256, valueTypes, 4);
---
> ValueType valueTypes[] = {TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE};
> OrderedSeparatorLineParser parser(" ", 256, valueTypes, 4, "#");
2 Likes

Outstanding! Thanks! I didn’t know about the optional header argument. Bravo!

1 Like

I am revisiting this to optimize and finalize the app for integrating a YSI EXO2 sonde. Here are some of my thoughts and questions…

  • I would like to setup the sonde to sample every 20 minutes for 30 seconds and transmit data out every 1 hour. How should I approach this? Should I program the sonde for that sampling period using YSI’s KOR software or will the flashed firmware (my code) be responsible for putting the sonde to ā€œsleepā€, ā€œwaking upā€ for sample events, recording data, and transmitting data?

  • I cannot figure out how to successfully send a message to the sonde. I would like to have code written that sends a command to initiate the wiper process before the sonde starts its sample period. Again, would this be hard coded or part of the sonde’s deployment setup?

/*
 * This is a lightweight example Dev Kit application (v.0.6.0) to integrate a YSI EXO2 Sonde 1Hz.
 * Sonde Paramters: temp, conductivity, DO, pH
 * '#' is user prompt
 * Commands are not case sensitive
 * Only spaces are recognized as delimeters
 * A command is terminated by a <CR>
 * Minimum time from power up to valid readings is 19 seconds
 * Example line: # 21.853 0.790 0.00 7.58 
*/

#include "user_code.h" // Contains your project-specific logic, constants, and function declarations (e.g. sampling interval, EXO2 integration details).
#include "LineParser.h" // Utilities to parse incoming lines of data—important for interpreting the ASCII output from the EXO2 Sonde.
#include "OrderedSeparatorLineParser.h" // Utilities to parse incoming lines of data—important for interpreting the ASCII output from the EXO2 Sonde.
#include "array_utils.h" // Utility functions for working with arrays (e.g., calculating min, max, mean, etc.).
#include "avgSampler.h" // Manages statistical sampling—likely used to average sensor readings over your aggregation interval.
#include "bm_network.h" // Interfaces with the Bristlemouth network stack for communication.
#include "bm_printf.h" // Provides a custom printf-like function tailored to Bristlemouth or embedded debug printing.
#include "bm_pubsub.h" // Handles the publish-subscribe messaging model over Bristlemouth. Useful for passing sensor data up to the Spotter buoy.
#include "bristlefin.h" // Possibly hardware abstraction for the Bristlefin board (e.g., managing I/O, power to peripherals like EXO2).
#include "bsp.h" // Core hardware-specific definitions, like pins, peripherals, power control.
#include "debug.h" // Debugging utilities—can include macros to print, assert, log, etc.
#include "lwip/inet.h" // Lightweight IP stack for network address utilities—may not be critical unless you're doing low-level networking.
#include "payload_uart.h" // Likely manages the UART connection to the EXO2 Sonde via the Bristlemouth payload interface.
#include "sensors.h" // A general sensor abstraction layer—might include functions for polling or parsing EXO2 sensor data.
#include "stm32_rtc.h" // Real-Time Clock interface for timekeeping. Important for timestamping samples.
#include "task_priorities.h" // Defines FreeRTOS (or similar) task priorities so your code runs in the correct order and with proper scheduling.
#include "uptime.h" // Likely provides system uptime tracking for logs or diagnostics.
#include "usart.h" // Universal Synchronous/Asynchronous Receiver-Transmitter—used to configure UART communication.
#include "util.h" // Miscellaneous utility functions—could include math helpers, string manipulation, timing, etc.



// float: Typically 4 bytes (32 bits); ~7 decimal digits of precision.
// double: Typically 8 bytes (64 bits); ~15-16 decimal digits of precision.
// So, using double roughly doubles the memory used compared to float for each number, but gives you more precise and accurate representation of fractional numbers.
// uint16_t: "unsigned" means it can only represent non-negative numbers (0 and positive integers); "16-bit" means it uses 16 bits (2 bytes) of memory to store the number allowing values from 0 up to 65,535 (2¹⁶ - 1).
// sample_count: is the name of the variable (or struct field) and stores how many samples have been collected or aggregated.
typedef struct {
    uint16_t sample_count; 
    double temp_min;
    double temp_max;
    double temp_mean;
    double temp_stdev;
    double cond_min;
    double cond_max;
    double cond_mean;
    double cond_stdev;
    double dos_min;
    double dos_max;
    double dos_mean;
    double dos_stdev;
    double ph_min;
    double ph_max;
    double ph_mean;
    double ph_stdev;
} __attribute__((__packed__)) exoData_t; // This tells the compiler not to insert padding between structure members.

#define LED_ON_TIME_MS 20 // The LED stays on for 20 milliseconds.
#define LED_PERIOD_MS 1000 // The LED will blink every 1000 ms (1 second).
#define DEFAULT_BAUD_RATE 9600 // Sets the serial baud rate to 9600—standard for many sensors, including YSI EXO2.
#define DEFAULT_LINE_TERM 13 // 13 corresponds to the ASCII Carriage Return (CR or \r)—used to detect end of command from EXO2.
#define BYTES_CLUSTER_MS 50 // Likely used for grouping incoming bytes during serial reads—bytes within 50ms of each other are considered part of the same "line."
#define EXO_AGG_PERIOD_MIN 5 // Averaging and aggregating EXO2 data every 5 minutes.
#define EXO_AGG_PERIOD_MS (EXO_AGG_PERIOD_MIN * 60 * 1000) // This converts the 5 minutes into milliseconds (300000 ms).
#define MAX_EXO_SAMPLES ((EXO_AGG_PERIOD_MS / 1000) + 10) // Calculates the maximum number of 1 Hz samples expected in the aggregation window: 5 minutes = 300 seconds → 300 sample; Adds 10 extra slots as a buffer for missed timing or startup lag.
#define WIPER_INTERVAL_MS (1 * 60 * 1000) // Sets the wipe interval to once every 60,000 ms (1 minute).
#define EXO_DATA_SIZE sizeof(exoData_t) // This lets you refer to the full size of the structure when allocating memory, sending over serial, or defining buffer sizes; Size: 2 + (16 * 8) = 130 bytes.

// In a global(file - level) context, like this, static limits the scope of each variable to this.cpp file only.
// Current serial baud rate and line termination for EXO2
static u_int32_t baud_rate_config = DEFAULT_BAUD_RATE;
static u_int32_t line_term_config = DEFAULT_LINE_TERM;

// In a global(file - level) context, like this, static limits the scope of each variable to this.cpp file only.
// Create a sampler object for each parameter that loves only in this file and is capable of collecting and averaging parameter values.
static AveragingSampler temp_data;
static AveragingSampler cond_data;
static AveragingSampler dos_data;
static AveragingSampler ph_data;

static int32_t ledLinePulse = -1; // A timer variable we can set to trigger a pulse on LED2 when we get payload serial data

char payload_buffer[2048]; // Declares a fixed-size character array named payload_buffer with room for 2048 bytes.

extern cfg::Configuration *userConfigurationPartition; // There’s a variable named userConfigurationPartition of type cfg::Configuration* defined somewhere else in the codebase — just not here.

ValueType valueTypes[] = {TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE, TYPE_DOUBLE}; // This creates an array of value types — specifically telling the parser what data type to expect for each field in a line of sensor output.
OrderedSeparatorLineParser parser(" ", 256, valueTypes, 4, "#"); // This creates an instance of OrderedSeparatorLineParser — used to split and interpret a line of sensor data.

void setup(void) {
    // These lines initialize circular or averaging buffers for storing sampled EXO2 data parameters.
    temp_data.initBuffer(MAX_EXO_SAMPLES);
    cond_data.initBuffer(MAX_EXO_SAMPLES);
    dos_data.initBuffer(MAX_EXO_SAMPLES);
    ph_data.initBuffer(MAX_EXO_SAMPLES);
    
    parser.init(); // Ensures the parser is in a clean and ready state before it begins processing input lines.
    PLUART::init(USER_TASK_PRIORITY); // Setup the UART – the on-board serial driver that talks to the RS232 transceiver.
    PLUART::setBaud(baud_rate_config); // Baud set per expected baud rate of the sensor.
    PLUART::setUseByteStreamBuffer(true); // Enable passing raw bytes to user app.
    PLUART::setUseLineBuffer(true); // Enable parsing lines and passing to user app. Warning: PLUART only stores a single line at a time. If your attached payload sends lines faster than the app reads them, they will be overwritten and data will be lost.
    PLUART::setTerminationCharacter((char)line_term_config); // Set a line termination character per protocol of the sensor.
    PLUART::enable(); // Turn on the UART.
    bristlefin.enableVbus(); // Ensures the EXO2 Sonde has power to operate.
    vTaskDelay(pdMS_TO_TICKS(5)); // After powering hardware (enableVbus() in this case), to give it time to stabilize before communication begins.
    bristlefin.enableVout(); // Calling enableVout() turns on this 12V voltage line so that connected hardware (e.g., EXO2) can receive power.
}

void loop(void) {
    
    static u_int32_t exoStatsTimer = uptimeGetMs(); // Initializes a timer variable that tracks how long the system has been running in MS.

    if ((u_int32_t)uptimeGetMs() - exoStatsTimer >= EXO_AGG_PERIOD_MS) { 
        
        exoStatsTimer = uptimeGetMs(); // reset the timer by storing the current system uptime
        double temp_mean = 0, temp_stdev = 0, temp_min = 0, temp_max = 0;   // Declare and initialize local variables to store statistics for each sensor parameter. All statistics start at zero for the new aggregation.
        double cond_mean = 0, cond_stdev = 0, cond_min = 0, cond_max = 0;
        double dos_mean = 0, dos_stdev = 0, dos_min = 0, dos_max = 0;
        double ph_mean = 0, ph_stdev = 0, ph_min = 0, ph_max = 0;
        uint16_t n_samples = 0; // Track how many samples were aggregated.
        
        // Checks if parameter.data has data in it. If so, calculate statistics. If zero, skip stats calculation block
        if (temp_data.getNumSamples()) { 
            temp_mean = temp_data.getMean();
            temp_stdev = temp_data.getStd(temp_mean);
            temp_min = temp_data.getMin();
            temp_max = temp_data.getMax();
        }
        
        if (cond_data.getNumSamples()) {
            cond_mean = cond_data.getMean();
            cond_stdev = cond_data.getStd(cond_mean);
            cond_min = cond_data.getMin();
            cond_max = cond_data.getMax();
        }
        
        if (dos_data.getNumSamples()) {
            dos_mean = dos_data.getMean();
            dos_stdev = dos_data.getStd(dos_mean);
            dos_min = dos_data.getMin();
            dos_max = dos_data.getMax();
        }
        
        if (ph_data.getNumSamples()) {
            ph_mean = ph_data.getMean();
            ph_stdev = ph_data.getStd(ph_mean);
            ph_min = ph_data.getMin();
            ph_max = ph_data.getMax();
        }
        
        n_samples = temp_data.getNumSamples(); // This captures the sample size of the data aggregation.
        
        temp_data.clear(); // These lines clear the stored data in each of the sampling buffers
        cond_data.clear();
        dos_data.clear();
        ph_data.clear();

        RTCTimeAndDate_t time_and_date = {}; // create a time/date variable using the RTCTimeAndDate_t struct type
        rtcGet(&time_and_date); // This gets the memory address of time_and_date so the function can fill it in with the current time and date.
        char rtcTimeBuffer[32]; // Declares a character array named rtcTimeBuffer that can hold up to 32 characters.
        rtcPrint(rtcTimeBuffer, &time_and_date); // Fills the rtcTimeBuffer with a formatted date/time string based on the contents of time_and_date.
        
        // Formats and prints data to a log file
        bm_fprintf(0, "payload_data_agg.log",
                   "tick: %llu, rtc: %s, n: %u, temp_min: %.4f, temp_max: %.4f, temp_mean: %.4f, temp_std: %.4f, "
                   "cond_min: %.4f, cond_max: %.4f, cond_mean: %.4f, cond_std: %.4f, "
                   "dos_min: %.4f, dos_max: %.4f, dos_mean: %.4f, dos_std: %.4f, "
                   "ph_min: %.4f, ph_max: %.4f, ph_mean: %.4f, ph_std: %.4f\n",
                   uptimeGetMs(), rtcTimeBuffer, n_samples,
                   temp_min, temp_max, temp_mean, temp_stdev,
                   cond_min, cond_max, cond_mean, cond_stdev,
                   dos_min, dos_max, dos_mean, dos_stdev,
                   ph_min, ph_max, ph_mean, ph_stdev);
        // Prints AGG data to the spotter console
        bm_printf(0,
                  "[exo-agg] | tick: %llu, rtc: %s, n: %u, temp_min: %.4f, temp_max: %.4f, temp_mean: %.4f, temp_std: %.4f, "
                  "cond_min: %.4f, cond_max: %.4f, cond_mean: %.4f, cond_std: %.4f, "
                  "dos_min: %.4f, dos_max: %.4f, dos_mean: %.4f, dos_std: %.4f, "
                  "ph_min: %.4f, ph_max: %.4f, ph_mean: %.4f, ph_std: %.4f",
                  uptimeGetMs(), rtcTimeBuffer, n_samples,
                  temp_min, temp_max, temp_mean, temp_stdev,
                  cond_min, cond_max, cond_mean, cond_stdev,
                  dos_min, dos_max, dos_mean, dos_stdev,
                  ph_min, ph_max, ph_mean, ph_stdev);
        // Prints AGG data to the serial console 
        printf("[exo-agg] | tick: %llu, rtc: %s, n: %u, temp_min: %.4f, temp_max: %.4f, temp_mean: %.4f, temp_std: %.4f, "
               "cond_min: %.4f, cond_max: %.4f, cond_mean: %.4f, cond_std: %.4f, "
               "dos_min: %.4f, dos_max: %.4f, dos_mean: %.4f, dos_std: %.4f, "
               "ph_min: %.4f, ph_max: %.4f, ph_mean: %.4f, ph_std: %.4f\n",
               uptimeGetMs(), rtcTimeBuffer, n_samples,
               temp_min, temp_max, temp_mean, temp_stdev,
               cond_min, cond_max, cond_mean, cond_stdev,
               dos_min, dos_max, dos_mean, dos_stdev,
               ph_min, ph_max, ph_mean, ph_stdev);
        
        // Create a struct instance tx_exo packed with all the aggregated data statistics, ready to be used for sending or logging. This structure exoData_t was defined int he Core data Structure section.
        exoData_t tx_exo = {
            .sample_count = n_samples,
            .temp_min = temp_min, .temp_max = temp_max, .temp_mean = temp_mean, .temp_stdev = temp_stdev,
            .cond_min = cond_min, .cond_max = cond_max, .cond_mean = cond_mean, .cond_stdev = cond_stdev,
            .dos_min = dos_min, .dos_max = dos_max, .dos_mean = dos_mean, .dos_stdev = dos_stdev,
            .ph_min = ph_min, .ph_max = ph_max, .ph_mean = ph_mean, .ph_stdev = ph_stdev };
        
        uint8_t tx_data[EXO_DATA_SIZE] = {}; // This tx_data array is a raw byte buffer designed to hold the serialized data of your exoData_t struct, so it can be sent over telemetry.
        memcpy(tx_data, (uint8_t *)(&tx_exo), EXO_DATA_SIZE); // Serializes the struct into a flat byte buffer, which can then be sent via telemetry.
        
        // This code block tries to send the prepared EXO2 data over the Spotter buoy network, then prints whether the send was successful or not on the serial console.
        if (spotter_tx_data(tx_data, EXO_DATA_SIZE, BM_NETWORK_TYPE_CELLULAR_IRI_FALLBACK)) {
            printf("%llut - %s | Successfully sent Spotter transmit data request\n", uptimeGetMs(), rtcTimeBuffer);
        } else {
            printf("%llut - %s | Failed to send Spotter transmit data request\n", uptimeGetMs(), rtcTimeBuffer);
        }
    }

  // This checks for a trigger set by ledLinePulse when data is received from the payload UART. Each time this happens, we pulse LED2 Green.
  static bool led2State = false;
  if (!led2State && ledLinePulse > -1) {
    bristlefin.setLed(2, Bristlefin::LED_GREEN);
    led2State = true;
  }
  // If LED2 has been on for LED_ON_TIME_MS, turn it off.
  else if (led2State && ((u_int32_t)uptimeGetMs() - ledLinePulse >= LED_ON_TIME_MS)) {
    bristlefin.setLed(2, Bristlefin::LED_OFF);
    ledLinePulse = -1;
    led2State = false;
  }

  static u_int32_t ledPulseTimer = uptimeGetMs();
  static u_int32_t ledOnTimer = 0;
  static bool led1State = false;

  // Turn LED1 on green every LED_PERIOD_MS milliseconds.
  if (!led1State && ((u_int32_t)uptimeGetMs() - ledPulseTimer >= LED_PERIOD_MS)) {
    bristlefin.setLed(1, Bristlefin::LED_GREEN);
    ledOnTimer = uptimeGetMs();
    ledPulseTimer += LED_PERIOD_MS;
    led1State = true;
  }
  // If LED1 has been on for LED_ON_TIME_MS milliseconds, turn it off.
  else if (led1State && ((u_int32_t)uptimeGetMs() - ledOnTimer >= LED_ON_TIME_MS)) {
    bristlefin.setLed(1, Bristlefin::LED_OFF);
    led1State = false;
  }

  // A timer is used to try to keep clusters of bytes (say from lines) in the same output.
  static int64_t readingBytesTimer = -1;
  // Note - PLUART::setUseByteStreamBuffer must be set true in setup to enable bytes.
  if (readingBytesTimer == -1 && PLUART::byteAvailable()) {
    // Get the RTC if available
    RTCTimeAndDate_t time_and_date = {};
    rtcGet(&time_and_date);
    char rtcTimeBuffer[32];
    rtcPrint(rtcTimeBuffer, &time_and_date);
    printf("[payload-bytes] | tick: %" PRIu64 ", rtc: %s, bytes:", uptimeGetMs(), rtcTimeBuffer);
    // not very readable, but it's a compact trick to overload our timer variable with a -1 flag
    readingBytesTimer = (int64_t)((u_int32_t)uptimeGetMs());
  }
  while (PLUART::byteAvailable()) {
    readingBytesTimer = (int64_t)((u_int32_t)uptimeGetMs());
    uint8_t byte_read = PLUART::readByte();
    printf("%02X ", byte_read);
  }
  if (readingBytesTimer > -1 &&
      (u_int32_t)uptimeGetMs() - (u_int32_t)readingBytesTimer >= BYTES_CLUSTER_MS) {
    printf("\n");
    readingBytesTimer = -1;
  }

  if (PLUART::lineAvailable()) {
    if (readingBytesTimer > -1) {
        printf("\n");
        readingBytesTimer = -1;
    }
    uint16_t read_len = PLUART::readLine(payload_buffer, sizeof(payload_buffer));

    RTCTimeAndDate_t time_and_date = {};
    rtcGet(&time_and_date);
    char rtcTimeBuffer[32] = {};
    rtcPrint(rtcTimeBuffer, NULL);
    bm_fprintf(0, "exo_raw.log", "tick: %" PRIu64 ", rtc: %s, line: %.*s\n", uptimeGetMs(), rtcTimeBuffer, read_len, payload_buffer); // Save raw data to exo_raw.log on SD card
    bm_printf(0, "[exo] | tick: %" PRIu64 ", rtc: %s, line: %.*s", uptimeGetMs(), rtcTimeBuffer, read_len, payload_buffer); // Print line to spotter console
    printf("[exo] | tick: %" PRIu64 ", rtc: %s, line: %.*s\n", uptimeGetMs(), rtcTimeBuffer, read_len, payload_buffer); // Print line to serial console

    ledLinePulse = uptimeGetMs();
    // Incoming line from the Sonde is parsed. Extracted values are added to sample buffers(e.g., temp_data, cond_data, etc.).
    if (parser.parseLine(payload_buffer, read_len)) {
        double temp = parser.getValue(0).data.double_val;
        double cond = parser.getValue(1).data.double_val;
        double dos = parser.getValue(2).data.double_val;
        double ph = parser.getValue(3).data.double_val;

        temp_data.addSample(temp);
        cond_data.addSample(cond);
        dos_data.addSample(dos);
        ph_data.addSample(ph);

        printf("count: %u/%d\n", temp_data.getNumSamples(), MAX_EXO_SAMPLES); // Print the sample count to the serial console.

    } else {
        printf("Error parsing line!\n");
        return;
    }

    if (temp_data.getNumSamples() >= MAX_EXO_SAMPLES) {
        printf("ERR - No more room in exo reading buffer, already have %d readings!\n", MAX_EXO_SAMPLES);
        return;
    }
  }

}  

I might be able to address this question:

I would like to setup the sonde to sample every 20 minutes for 30 seconds and transmit data out every 1 hour. How should I approach this? Should I program the sonde for that sampling period using YSI’s KOR software or will the flashed firmware (my code) be responsible for putting the sonde to ā€œsleepā€, ā€œwaking upā€ for sample events, recording data, and transmitting data?

Sofar’s preferred way to do sample periods and duty cycles is to cycle the entire Smart Mooring bus using the Bridge Conroller. This has a few advantages:

  • Saves power by having the entire system off when not sampling
  • Adjustable configurations that can be set over the terminal without adjusting the code on the mote
  • Can be adjusted over Satellite or LTE by contacting support@sofarocean.com which can help during remote testing or seasonal deployments (e.g. less sun in Alaska in the winter…)

One way to approach this…er…approach would be:

  1. Find out how much time the Exo needs to do all the things it needs to do such as power on, self check, run all the different sampling things, and retrieve the data over serial (with the necessary RS232 adapter board they sell) or SDI12. Don’t forget the wiper operations on any optical sensors! Let’s call that time hypothetically, 6 minutes.
  2. Think about how often you wan to sample like this. Let’s say every hour.
  3. You code should be set up to leave alone the bridge configurations as we’ll be adjusting these manually in this approach…but it should run when it starts, wait the appropriate amount of time (6 minutes + 1 minute to be safe) and then trigger the telemetry, SD card saves, and any logs you want to log.
  4. Manually set the bridge configurations for every, let’s say 8 minutes (6 for the Sonde to do it’s thing, one minute for you code to do all it’s things, and 1 more minute for some unforseen issues), every hour. A more confident firmware programmer can probably suggest more educated timings. To manually set the bridge use these commands on the terminal on SPOTTER:
bridge cfg set 0 s u bridgePowerControllerEnabled 1
bridge cfg set 0 s u sampleIntervalMs 3600000      # Sets interval to 1 hour (ms)
bridge cfg set 0 s u sampleDurationMs 480000      # Sets on-duration for 8 min (ms)
bridge cfg commit 0 s

You can give this a try with just the regular ā€œsimpleā€ example in the guide as that would provide internal data at this cadence.

I don’t know how to answer the next question:

I cannot figure out how to successfully send a message to the sonde. I would like to have code written that sends a command to initiate the wiper process before the sonde starts its sample period. Again, would this be hard coded or part of the sonde’s deployment setup?

@umakat , do you have any ideas on what @cmchrist can do here that’s pretty straightforward? Does the Exo do this on it’s own or do we need to send these commands as part of the code?

@cmchrist I’d also take a minute to follow @umakat’s outsanding documentation on her effort to use SDI12 to talk to the Exo3.

Hi @cmchrist!

My understanding is that every time the sonde makes a measurement, either via internal sampling or by a command, the wiper will begin a wipe and then a measurement is taken.

In the EXO3s integration, I send a command via SDI-12 communication interface to ask the EXO3s to make a measurement. It begins a wipe, and ~62 seconds later sends over one set of readings. We haven’t tested an internal sampling scenario yet.

I think the RS232 command to get the EXO2 to make a measurement is different from the SDI-12 command. But on the EXO manual page 43, they list a RS232 extended command twipeb to start a wipe. Have you had a chance to try this command and see if it works for your case?

1 Like