Simple_payload_example : how does it work?

Application Dev
1

I’m trying to figure out how data is read from the RS-232 device in the simple_payload_example. First some preliminaries:

Link to simple_payload_example/user_code/user_code.cpp

The simple_payload_example shows how to setup, read , and log line-oriented data from a RS-232 sensor…a very common use case for oceanographic sensors. By line-oriented data, I mean typically ASCII-formated data the ends with a newline (\n, 0x0A). Each ASCII line of data is logged opaquely on Spotter SD card using spotter_log(). With a small addition of code (calling spotter_tx_data()), each line of data can be transmitted via the Spotter buoy.

In my case, every 15 min, the instrument simply sends out a comma-delimited string like this (terminated with a newline) :
SSPHOX02001,08/12/2020 12:26:54, 4, 0000, 5.3506, 7.9147, 7.8729, -0.857475, -0.900347, 8.6681, 0.061, 33.7431, 3.26786, 6.889, -0.5, 14.5\n

The user_code.cpp for any mote has two entry points: setup() and loop(). For the simple_payload_example, the setup()code is used to one-time stuff like grabbing some CFG data, setting the UART mode and RS-232 parameters accordingly, enabling the ByteStream and Line buffers, and setting the TerminationCharacter (0x0A). Depending on the sensor’s power needs, setup() may turn on 12V, 5V, and/or 3V power. Note in my case, the sensor has its own batteries. (The setup() code for my application only enables the 12V power.)

loop() is where the good real-time data stuff happens. FreeRTOS calls loop() continuosly. Beyond some LED twiddling (which I’ll ignore for now), three major things happen:

  1. A small while() loop gobbles up bytes.
  // Read a cluster of bytes if available
  // -- A timer is used to try to keep clusters of bytes (say from lines) in the same output.
  static int64_t readingBytesTimer = -1;
  // Note - PLUART::setUseByteStreamBuffer must be set true in setup to enable bytes.
  if (readingBytesTimer == -1 && PLUART::byteAvailable()) {
    // Get the RTC if available
    RTCTimeAndDate_t time_and_date = {};
    rtcGet(&time_and_date);
    char rtcTimeBuffer[32];
    rtcPrint(rtcTimeBuffer, &time_and_date);
    printf("[payload-bytes] | tick: %llu, rtc: %s, bytes:", uptimeGetMs(), rtcTimeBuffer);
    // not very readable, but it's a compact trick to overload our timer variable with a -1 flag
    readingBytesTimer = (int64_t)((u_int32_t)uptimeGetMs());
  }
  while (PLUART::byteAvailable()) {
    readingBytesTimer = (int64_t)((u_int32_t)uptimeGetMs());
    uint8_t byte_read = PLUART::readByte();
    printf("%02X ", byte_read);
  }
  if (readingBytesTimer > -1 &&
      (u_int32_t)uptimeGetMs() - (u_int32_t)readingBytesTimer >= BYTES_CLUSTER_MS) {
    printf("\n");
    readingBytesTimer = -1;
  }
  1. Check to see if a line has been read, i.e., do I have in my Line buffer a string terminated by 0x0A? If so, read it.
// Read a line if it is available
// Note - PLUART::setUseLineBuffer must be set true in setup to enable lines.
if (PLUART::lineAvailable()) {
// Shortcut the raw bytes cluster completion so the parsed line will be on a new console line
  if (readingBytesTimer > -1) {
    printf(“\n”);
    readingBytesTimer = -1;
  }
  uint16_t read_len = PLUART::readLine(payload_buffer, sizeof(payload_buffer));

  // Get the RTC if available
  RTCTimeAndDate_t time_and_date = {};
  rtcGet(&time_and_date);
  char rtcTimeBuffer[32];
  rtcPrint(rtcTimeBuffer, &time_and_date);
  1. Log and, in my case, transmit the data
  // Print the payload data to a file, to the bm_printf console, and to the printf console.
  spotter_log(0, “payload_data.log”, USE_TIMESTAMP, “tick: %llu, rtc: %s, line: %.*s\n”,
  uptimeGetMs(), rtcTimeBuffer, read_len, payload_buffer);
  spotter_log_console(0, “[payload] | tick: %llu, rtc: %s, line: %.*s”, uptimeGetMs(),
  rtcTimeBuffer, read_len, payload_buffer);
  printf(“[payload-line] | tick: %llu, rtc: %s, line: %.*s\n”, uptimeGetMs(),
rtcTimeBuffer, read_len, payload_buffer);

  // Transmit over Spotter celluar or Iridium SBD fallback.
  if (spotter_tx_data(payload_buffer, sizeof(payload_buffer), BmNetworkTypeCellularIriFallback)) {
    printf("%llut - %s | Sucessfully sent Spotter transmit data request\n", uptimeGetMs(), rtcTimeBuffer);
  } else {
    printf("%llut - %s | Failed to send Spotter transmit data request\n", uptimeGetMs(), rtcTimeBuffer);
  }
}

Here are my questions:

  • What is the purpose of Step 1 ( while loop with readByte() ) ?
  • Why doesn’t Step 1 always gobble up all of the bytes, i.e. why is there any data left in the Line buffer?
  • Is the byte buffer (readByte()) and LineBuffer (readLine()) separate? If so, does that mean there are two copies of the data somewhere?
  • Why even have the Byte buffer if you’re doing line-oriented data capture?

In short, how do I ever end up with any data in the payload_buffer() ? Obviously, since I’m getting data, this code works. But I don’t really understand what it’s doing.

Please explain….

Thanks!

2

Hello @sbs-erehm and welcome to the forums!

I can answer that question for you and give you some clarity!
The code that handles all of the UART communications is payload_uart.cpp.
Within this module, the UART handling can be configured to use a stream buffer (which is a fancy FreeRTOS buffer that internally handles multi-thread access), and/or use the line buffer. The functions to enable both of these are respectively:

  • void setUseByteStreamBuffer(bool enable);
  • void setUseLineBuffer(bool enable);

A separate buffer is used for both of these functionalities.
The _user_line struct located here, and user_byte_stream_buffer located here.

When the following block is called:

while (PLUART::byteAvailable()) {
    readingBytesTimer = (int64_t)((u_int32_t)uptimeGetMs());
    uint8_t byte_read = PLUART::readByte();
    printf("%02X ", byte_read);
  }

The code is reading from the stream buffer (user_byte_stream_buffer).

What is the purpose of Step 1 ( while loop with readByte() ) ?

This is strictly to print the raw bytes to the serial console.

When the following is invoked:

if (PLUART::lineAvailable()) {
// Shortcut the raw bytes cluster completion so the parsed line will be on a new console line
  if (readingBytesTimer > -1) {
    printf(“\n”);
    readingBytesTimer = -1;
  }
  uint16_t read_len = PLUART::readLine(payload_buffer, sizeof(payload_buffer));
}

The data is being read from the line buffer (_user_line).

Is the byte buffer (readByte()) and LineBuffer (readLine()) separate? If so, does that mean there are two copies of the data somewhere?

Yes there are two copies of the data.

Why even have the Byte buffer if you’re doing line-oriented data capture?

In practical use cases, only one of the options are worthwhile.
This example showcases both of the methods available! One that is printing the raw bytes to the console and the other is used to actually send the data remotely.

You can imagine that a sensor (or maybe a co-processor such as a Raspberry Pi) might have raw data that is not necessarily delimited by a single character, or data that might need to be handled on the fly as the bytes are being ingested, in that case the stream buffer is worth using. It handles each byte individually.

A sensor that sends data with a character delimiter that is always the same and does not need low-latent ingestion of individual byte would benefit from using the line buffer. This allows for the data to be handled all at once and is useful for the application of forwarding the whole string collected remotely to shore!

Hopefully that answers your question!
Let me know if I can help with anything else!

1 Like
3

Thanks for the clear explanation!

I would suggest adding some of this same context (e.g., “In practical use cases, only one of the options are worthwhile….”) to the DevKit example "Bristlemouth Dev Kit Guide: Integrating an RS232 Serial Sensor”…and possibly the simple_payload_example comments.