Spotter Cellular/Sat transmission queue constraints and documentation

Application Dev
1

Hi,

I am currently trying to use the bm devkit connected to spotter in order to send custom sensor data from my sensor (connected to the devkit via uart) through cellular and retrieve it via the spotter api.

Currently my sensor is sending a data packet of 272 bytes every 5 seconds to the devkit. The devkit is running custom code (I have used some of the serial_payload_example code and added a spotter_tx_data call to send this data as soon as it is received from the sensor).
I can see from the API calls that the data was being sent via cellular every 5 seconds for a period of 15 minutes (from 2024-10-02T14:20:55.000Z to 2024-10-02T14:34:45.000Z) after which there has been no data sent. The spotter device also started behaving strangely after this (lost GPS signal although it wasn’t moved; the signal LED turned red).
Note: I’m not sure if this is due to the queue being full or that it just coincidentally lost the cellular and GPS signal at this time.

Meanwhile, I was simultaneously monitoring the spotter via serial terminal and saw that the messages were getting queued in MS_Q_LEGACY until the id hit 205 (when I started this run, the id was at 11), following which it said that the queue is full and is unable to submit the message to the sat/cell queue.

I then disconnected my sensor and just let the devkit and spotter run normally (without any payload input) and the GPS and cellular signal was regained after about 8 minutes. Some of the previously queued data was sent and I was able to see it in the API call. After this, I reconnected my sensor and messages started queuing again for a while before the queue was full again.

I would like to know if there is any documentation for the queuing process on the spotter and what the limits are for the same.
Additionally, it would be helpful to know your recommendations when sending such high frequency data via the cellular w iridium fallback option (as it is the only option currently supported by the spotter API)

Thanks,
Gautam

2

Hi,

Just wanted to post an update with some new findings.
I ran the system again (sensor connected to devkit via uart sending data every 5 seconds; devkit connected to spotter sending a spotter_tx_data call when data is received from sensor) for about 1 hour.
This time, there was no issues on the spotter. It was able to continuously queue messages and send them via cellular. About 680 messages had been successfully sent and were visible on the spotter API call.

After an hour, I saw the same queue full message again.
I’m thinking what happened yesterday (described in the original post) was probably due to a loss in cellular connection. I tried to reconfirm if that was the issue this time as well, but the LEDs had turned off and I wasn’t able to find a command to check the cellular status on spotter.

The questions I still have are:

  1. What is the queue size/limit on the spotter, in case the cellular connection is lost?
  2. Is there any documentation available regarding spotter’s queuing and cellular transmission process? Or is there someone I can talk to in order to get more clarity about this?
  3. Is there any way to query the spotter device for cellular connectivity status (either from an external microcontroller connected to the devkit or from devkit itself)
  4. Is there a way to check cellular connection logs on the spotter?
  5. Dos and donts while sending data at high frequencies using cellular w iridium fallback
3

Hi Gautam,

Great questions. Researching for authoritative answers to some of them will take time, but I wanted to give you a quick reply with some helpful info I know off the top of my head.

There’s plenty of helpful documentation for Spotter. Here are a few links.

Regarding your questions:

  1. Don’t know. TBD.
  2. Links above are helpful, but this conversation on the forum is the best place, so the whole community can learn and chime in with their tips.
  3. No way to query cell status right now — feature request is already in my team’s backlog.
  4. Logs, see links above.

As you’ve found, the throughput is highly dependent on the environment. We make every effort to send Spotter’s internal data as efficiently as possible, sending summarized stats on the order of hourly. You’re pushing the boundaries of what’s possible right now by sending every 5 minutes — and that’s great! You may have more success though (here’s a “do”) by processing more data in the embedded code so that telemetry can be smaller aggregated statistics sent less frequently. However, depending on your use case that may not be reasonable.

In the backlog for the Sofar Ocean backend software team is the end-to-end functionality for the cellular-only queue. Lots of people want it. We want it! But in our ruthless prioritization, it hasn’t risen to the top of the priority list yet. Until that works, be mindful that your Spotter can lose cell signal and start sending by satellite, which is more expensive.

Hope that helps! I (or someone else) will try to answer the queue size question within the next couple weeks.

1 Like
4

Hello Gautam!

The queue size currently is 32 total messages for Iridium and Cellular.
If you are seeing MS_Q_LEGACY this means that there is no cell connection and you have fallen back to iridium.
Hope that provides some insight!

5

Thanks for the information @matt001k!

I am a bit confused though. I was monitoring spotter logs while it was running (and had a cellular connection active as seen on the spotter dashboard site as well as the signal LED being solid green)

I have attached the logs below:
2024-10-03T14:57:48.140Z [ERR] [INFO] Cellular SignalErrorState changed from CONNECTING to OK
2024-10-03T14:57:48.156Z [MS] [DEBUG] Has message 332 expired? queuedTime: 6454458, now64: 6489320, remaining: 86365138
2024-10-03T14:57:48.156Z [MS] [DEBUG] Has message 332 expired? queuedTime: 6454458, now64: 6489321, remaining: 86365137
2024-10-03T14:57:48.156Z [MS] [DEBUG] Requeueing unexpired message 332 to queue(6)
2024-10-03T14:57:48.156Z [MS] [DEBUG] Has message 332 expired? queuedTime: 6454458, now64: 6489322, remaining: 86365136
2024-10-03T14:57:48.160Z [ERR] [INFO] Cellular SignalErrorState changed from OK to CONNECTING
2024-10-03T14:57:48.160Z [MS] [DEBUG] Sending legacy messages to Notecard.
2024-10-03T14:57:48.179Z [MS] [DEBUG] Notecard is 2.000000 pct full.
2024-10-03T14:57:48.179Z [MS] [INFO] Queuing message 332 20004550
2024-10-03T14:57:49.015Z [MS] [DEBUG] Has message 333 expired? queuedTime: 6459734, now64: 6490181, remaining: 86369553
2024-10-03T14:57:49.015Z [MS] [INFO] Queuing message 333 20004600
2024-10-03T14:57:49.554Z [MS] [DEBUG] Has message 334 expired? queuedTime: 6465018, now64: 6490718, remaining: 86374300
2024-10-03T14:57:49.554Z [MS] [INFO] Queuing message 334 20016570
2024-10-03T14:57:50.093Z [MS] [DEBUG] Has message 335 expired? queuedTime: 6470301, now64: 6491257, remaining: 86379044
2024-10-03T14:57:50.093Z [MS] [INFO] Queuing message 335 2000F988
1727967470.269 53bcc703ce1e1390, [payload] | tick: 81242074, rtc: 2024-10-03T14:57:49.367, line: 2024-09-26T13:53:02,Avg Dark,2829389,Avg Blue,10326002,Avg Green,10095587,Temperature,22.25,Absorbance Blue,-0.002,Absorbance Green,0,R Ratio,-3.341,pH,-1,Ref Avg Dark,2255212,Ref Avg Blue,-11047,Ref Avg Green,12890720,Ref Absorbance Blue,nan,Ref Absorbance Green,-0.001
Current bit offset: 230
Rounded bit offset: 232
New bit offset: 2400
2024-10-03T14:57:50.296Z [MS] [INFO] Added message(id: 339 len: 300) to queue MS_Q_LEGACY: (4)!
Message: DE 66 FE B0 EE D9 FF A8 02 25 51 4B EE FD 1A 5E 0F 38 78 4E 41 4E F3 1C 0C BD 40 8A 7C 32 30 32 34 2D 30 39 2D 32 36 54 31 33 3A 35 33 3A 30 32 2C 41 76 67 20 44 61 72 6B 2C 32 38 32 39 33 38 39 2C 41 76 67 20 42 6C 75 65 2C 31 30 33 32 36 30 30 32 2C 41 76 67 20 47 72 65 65 6E 2C 31 30 30 39 35 35 38 37 2C 54 65 6D 70 65 72 61 74 75 72 65 2C 32 32 2E 32 35 2C 41 62 73 6F 72 62 61 6E 63 65 20 42 6C 75 65 2C 2D 30 2E 30 30 32 2C 41 62 73 6F 72 62 61 6E 63 65 20 47 72 65 65 6E 2C 30 2C 52 20 52 61 74 69 6F 2C 2D 33 2E 33 34 31 2C 70 48 2C 2D 31 2C 52 65 66 20 41 76 67 20 44 61 72 6B 2C 32 32 35 35 32 31 32 2C 52 65 66 20 41 76 67 20 42 6C 75 65 2C 2D 31 31 30 34 37 2C 52 65 66 20 41 76 67 20 47 72 65 65 6E 2C 31 32 38 39 30 37 32 30 2C 52 65 66 20 41 62 73 6F 72 62 61 6E 63 65 20 42 6C 75 65 2C 6E 61 6E 2C 52 65 66 20 41 62 73 6F 72 62 61 6E 63 65 20 47 72 65 65 6E 2C 2D 30 2E 30 30 31 0D
2024-10-03T14:57:50.304Z [BM_TX] [INFO] Submitted spotter/transmit-data message to sat/cell queue, Len: 272

I have always seen the messages being submitted to the MS_Q_LEGACY in all my test runs.
Is there something wrong in my call to spotter_tx_data maybe?

// Print the payload data to a file, to the bm_printf console, and to the printf console.
    bm_fprintf(0, "payload_data.log", "tick: %llu, rtc: %s, line: %.*s\n",
               uptimeGetMs(), rtcTimeBuffer, read_len, payload_buffer);
    bm_printf(0, "[payload] | tick: %llu, rtc: %s, line: %.*s", uptimeGetMs(),
              rtcTimeBuffer, read_len, payload_buffer);
    printf("[payload-line] | tick: %llu, rtc: %s, line: %.*s\n", uptimeGetMs(),
           rtcTimeBuffer, read_len, payload_buffer);

    ledLinePulse = uptimeGetMs(); // trigger a pulse on LED2

    if (spotter_tx_data(payload_buffer, read_len, BM_NETWORK_TYPE_CELLULAR_IRI_FALLBACK)) {
      printf("%llut - %s | Sucessfully sent Spotter transmit data request\n", uptimeGetMs(),
             rtcTimeBuffer);
    } else {
      printf("%llut - %s | Failed to send Spotter transmit data request\n", uptimeGetMs(),
             rtcTimeBuffer);
    }
6

Ahh apologies, after reviewing the code a bit further, that does look correct, items queued in that specific queue will transmit on Cell with Iridium fallback. There are other queues associated in the same file that are in relation to different functionality (the other one being MS_Q_CELLULAR_ONLY), please disregard that last statement!

7

Thanks for the confirmation @matt001k.
I have seen the MS_Q_CELLULAR_ONLY being used for sending notecard messages.

I’m curious if there is a way for me to use that queue to send my data instead of relying on MS_Q_LEGACY.

8

The “legacy” queue (poorly named since it’s the main queue for almost all messages) is the only one usable from the Bristlemouth network right now. The cellular-only queue works on the embedded side, but more backend server work is required to make it all the way to the Sofar API to be accessible to you. That’s what I was referring to with this paragraph:

1 Like