Camera payload goals and sending images over cellular

Application Dev
1

HI! I’m back with updates from the UCSB capstone team.

To recap for anyone else this might help, the project goal given to us by our sponsors was to add a camera as a payload on the Bristlemouth system and establish a video livestream to be viewed onshore. We pretty quickly realized that’s not a realistic goal, so as of now our tentative goals are (roughly in order of what we think is most doable):

  • Saving photo and/or video data to the spotter’s SD card, and sending an alert over cellular when the SD card is nearly full.
  • Motion detection to tell the camera when something interesting is in the frame (when to start/stop recording)
  • Computer vision to count how many shark or fish looking things come in frame, and sending that data with timestamps
  • Sending a photo over the spotter’s cellular connection. This seems like a stretch, but we really don’t want to give up on it since it’s the closest goal to what our sponsors gave us.

A lot of these goals will be challenges to do with programming our microcontroller more than they will be challenges with Bristlemouth, so they’re not focused on in this post. Mostly here I’ll be talking about trying to send an image over cellular.

So far, we have a USB webcam and a Raspberry Pi 4 that we are using together as a payload connected to the dev board. We have established that we are able to power the Raspberry Pi from the dev board using a buck converter (thanks @NickRaymomd for the tip. We have been following your progress :wink:). We are also able to control the webcam settings and take photo and video from the cam, saving them to the RPi. The bigger challenge starts now as we are trying to integrate it into Bristlemouth.

The first proof of concept was to send simple data from the RPi and read it from the Dev Board console. We started by following the guide Bristlemouth Dev Kit Guide: Integrating an RS232 Serial Sensor. We flashed the dev board with the exact app firmware provided in that guide, and connected the raspberry pi serially to the dev board using UART tx and rx. I pulled together a short script on the raspberry pi to send single byte of integers incrementing every second, and without changing any of the dev board app code at all we were able to read the bytes from the dev board console. Success so far!

This brings me to my first set of questions for the Bristlemouth admin:

  • Uart is clearly not the ideal communication method for sending bulky visual data, but from what I have seen uart and rs232 are the only ones that have been developed so far. Do you have any other methods (SPI? dare I say USB? :eyes:) we should try before spending more time with UART?
  • In user_code.cpp for the rs232 app it indicates that the buffer size for received UART messages is 2048 bytes. Is there a strict upper limit for that? Also, can we use any standard baudrate for the mote? Currently it’s set to 9600, but I would love to crank that up a notch if that won’t cause problems.
  • Where can we look for resources on the uart commands? It’s hard to tell if these are standard compared to other cpp development environments or if they have been developed unique to bristlemouth.

For now, we’ve continued with our design assuming that UART is the best option and that 2048 bytes is the largest buffer that can be received from a payload. To get around that, I made a little program that will take any file (jpegs for now) and split its binary into individual binary files of an adjustable size (2048 bytes for now). Without fiddling with any camera settings, pixel width, or any fancy compression, the photos are coming out to be in the range 200 KBs, which translates to anywhere between 80 and 150 individual 2048 byte files. It’s a little awkward and getting a short video from the raspberry pi to the dev board will be significantly harder, but I think it’s doable. The plan is to send those packets to the dev board using code very similar to the RS232 example app and then route them one by one to the spotter.

The next step we will do is change the user code to send these buffers to the spotter and (for now) save them to files on the spotter’s SD card. Then we can remove the SD card and offload all those files to my laptop, where we can reassemble them into a single jpeg file to make sure that everything came through properly. It looks like Dev Kit Guide 5 should be a really good resource for that so we will continue using that as a reference. But… now comes the time for planning for how we will send this over cellular. Here’s my next chunk of questions:

  • Is there a size limit on data that is sent from the dev board to the spotter? If we wanted to, could we send big files (like GB range video files to be saved on the spotter’s SD) through that pipeline in one go?
  • I saw somewhere that there were issues sending data over cellular without iridium fallback. Right now I would rather not send data at all than rack up a fat Iridium bill. Do you have a workaround for cellular only sending?
  • I am not a pro regarding the API, but I know that it’s not set up to receive the type of data we want to be sending. Can we build on the API to have it receive raw binary files that are downloadable? Alternately can we direct outgoing data from the spotter to an alternate destination, like a separate website where they can be held? What do you see as the best way to accomplish this?? It seems like it will be a big challenge.
  • I’ve been piecing together bits of information regarding the cellular connection that I’ve read in various places and I’m not sure what’s relevant and what it means for us… First I saw that the cellular connection has no bandwidth restrictions. Then I saw that trying to send data larger than 340 bytes will cause an error. I think I also saw that we can only do uploads once every 5 minutes. Math is telling me that to send my 200KB of data I would need 590 packets sent over 49 hours… That’s not good… How much of this is true?

I think that concludes the questions and updates I have for right now. I’m hoping that y’all who know Bristlemouth’s ins and outs will have some insight on what is a good amount of a challenge versus what is an impossible challenge for us. We will keep updating as we continue! Thanks :smiley:

3 Likes
2

Hi there. Thanks for your detailed post.

Also thanks for your patience in a reply here. This post is quite long so it’s tough to get everything at once but we’ll try.

First thanks again for the effort you and your team are putting forward. We hope that this has and will continue to be an exciting journey into firmware development with regards to marine applications. Because Bristlemouth is brand new you guys are running into the end of the paved roads very quickly. We’ll be able to work through some of those things together and others might be too big or too difficult to scope right now so we’ll have to create some work-arounds or maybe make some concessions on what we can do today versus in the near future. We really appreciate your effort.

Right, let’s get into the questions:

Uart is clearly not the ideal communication method for sending bulky visual data, but from what I have seen uart and rs232 are the only ones that have been developed so far. Do you have any other methods (SPI? dare I say USB? :eyes:) we should try before spending more time with UART?

  • The Bristlemouth Mote supports UART, SPI, I2C, and has a native USB 2.0 interface.
  • There are example usages of SPI in the bm_protocol repository (see bm_soft app/bsp), but we don’t yet have a user guide for simple SPI applications. So making use of this today will require some familiarity with the STM32, how to configure a board support package in CubeMX, getting into the low level drivers, etc.
  • We have on the list to create a simple user guide + demo app for SPI, but can’t promise a specific timeline for that.
  • We have not dug into using native USB2.0 interface for data transfer yet. It currently is enumerated as a serial device for print functions in the application framework, so that would provide a starting point for a comfortable embedded developer.
  • All that being said - given the constraints in the rest of the system, we’d recommend starting with UART.

In user_code.cpp for the rs232 app it indicates that the buffer size for received UART messages is 2048 bytes. Is there a strict upper limit for that? Also, can we use any standard baudrate for the mote? Currently it’s set to 9600, but I would love to crank that up a notch if that won’t cause problems.

  • There isn’t a strict upper limit, you can use as much of the free memory in the app as wanted for that. Currently the hello_world app has ~140kB of free memory, but they could likely adjust the build settings to get up ~300kB.
  • You can crank it up! Max configurable baud rate is 833,333. Note - this of course will not work for RS232, but should be fine for raw UART if wire lengths are minimized.

Where can we look for resources on the uart commands? It’s hard to tell if these are standard compared to other cpp development environments or if they have been developed unique to bristlemouth.

  • One of the in-the-works documentation tasks are approachable developer references that will include details on arguments and behaviors of the various functions that are implemented. Currently, the code is the best reference / only documentation.
  • You could reference the implementation of the Mote’s payload UART interface handler, which is in src/lib/common/payload_uart.cpp.

Is there a size limit on data that is sent from the dev board to the spotter? If we wanted to, could we send big files (like GB range video files to be saved on the spotter’s SD) through that pipeline in one go?

  • Yes - the maximum size of any individual Bristlemouth message for the 10T1L PHY is 1,500 bytes. So the usable payload size is a little less than that: #define MAX_PAYLOAD_LEN (1500 - sizeof(struct ip6_hdr) - sizeof(struct udp_hdr)) See src/lib/middleware/middleware.cpp for more details on the interface between messaging middleware and network layer.
  • No one has implemented a File Transfer Protocol on Bristlemouth yet, so packetization for file transfer needs to be done directly in applications. Selecting and implementing an FTP on Bristlemouth is very high on the Core Developer feature request list, but we can’t offer a specific timeline quite yet.

I saw somewhere that there were issues sending data over cellular without iridium fallback. Right now I would rather not send data at all than rack up a fat Iridium bill. Do you have a workaround for cellular only sending?

  • We absolutely would not recommend sending images over Iridium unless you know exactly how much that will cost and it’s worth $$$ to you.

  • The issue with the cellular-only data transmission is that Sofar still needs implement the client APIs. The data makes it to the servers, but users can’t access it yet. This feature is on the roadmap, and is currently scheduled for April.

I am not a pro regarding the API, but I know that it’s not set up to receive the type of data we want to be sending. Can we build on the API to have it receive raw binary files that are downloadable? Alternately can we direct outgoing data from the spotter to an alternate destination, like a separate website where they can be held? What do you see as the best way to accomplish this?? It seems like it will be a big challenge.

  • There isn’t a way for users to program or configure the API to send data other than JSON text currently. Given the payload limits are currently hundreds of bytes, this isn’t a big deal right now. When we’re able to send larger files over the cellular system, we will need to implement binary file formats in the API as well.
  • For now, you could setup a server or client to poll the Sofar API for new data, or use the Webhooks-style POST forwarding feature described here so that new data will automatically be sent to your server for processing, concatenation and storage.

I’ve been piecing together bits of information regarding the cellular connection that I’ve read in various places and I’m not sure what’s relevant and what it means for us… First I saw that the cellular connection has no bandwidth restrictions. Then I saw that trying to send data larger than 340 bytes will cause an error. I think I also saw that we can only do uploads once every 5 minutes. Math is telling me that to send my 200KB of data I would need 590 packets sent over 49 hours… That’s not good… How much of this is true?

  • There currently are no data limit restrictions imposed by Sofar on the cellular data. This pertains to the data subscription and costs - not technical limitations.
  • The satellite uplink has a 340 byte per-message limit, which means a 310 byte max payload for a Bristlemouth user (~30 bytes of overhead for location and system info). So sending messages as “cellular with satellite fallback” you must stay within this limit. When the API access to cellular-only messages is implement in the next few months, then the size limitation is to stay within the 1,500 byte MTU for a Bristlemouth message. Note - the maximum transmission size for Spotter’s cellular telemetry is ~100kB, so once Bristlemouth supports a file transfer protocol larger cellular reports will be able to be sent.
  • The 5 minute limitation is for the periodic ocean weather data measured by Spotter. A Bristlemouth spotter/transmit-data message will trigger an immediate send over cellular. It may be wise to space a message out by some reasonable amount of time when possible to keep this from being a bottleneck. You can probably test this in the lab to see what works best for your message size but when you get into the field and there’s sporadic cellular connection this may break down.

Thanks again, we’re going to write up another quick post on cameras and other devices that need large file transmission shortly and I’ll make sure to link you to that.

We’re here to help so please let us know how we can best help.
~Zack and @evan and @vsowa12

1 Like
3

Edit: We’ll try to get some more info over to you ASAP (but for now excited about joining for the presentation)!