Trouble with flashing firmware

[Kaitlynyau]

Hello!
I was working through development kit guide: integrating a serial sensor but encountered some issues flashing the serial payload guide application. After following the steps outlined in how to update the Mote firmware, we did not recieve the ACK commands and had the system time out. In a previous attempt where the Hello world file from dev kit guide 3 was still present on the Spotter SD card we saw success in the firmware flash but did not see the correct file in the info command of the Dev Kit USB console. We were wondering if anyone else had this issue when trying to flash the firmware and had suggestions on how to move forward.
Also worth noting, the system light on our spotter is now solid red… We are wondering if this could be related/the cause of the issue, since we know that that can indicate SD card issues. What can we do to try and restore or reformat the SD card, and will that require wiping it and starting fresh?

[timjoh]

Hi @Kaitlynyau and welcome to the forum! Glad to hear you are working through the guides.

@Kaitlynyau

I was working through development kit guide: integrating a serial sensor but encountered some issues flashing the serial payload guide application. After following the steps outlined in how to update the Mote firmware, we did not recieve the ACK commands and had the system time out. In a previous attempt where the Hello world file from dev kit guide 3 was still present on the Spotter SD card we saw success in the firmware flash but did not see the correct file in the info command of the Dev Kit USB console. We were wondering if anyone else had this issue when trying to flash the firmware and had suggestions on how to move forward.

I have some follow-up questions which may lead us to some answers –

1. Was the serial payload guide application a bundle you downloaded from our documentation, or one you built & compiled yourself?

2. Have you had success re-flashing the mote with a firmware from one of the previous guides?

3. Which technique for flashing the firmware have you been using? Does it involve putting the firmware file onto the SD card, in the Spotter, and then issuing commands to flash from that file?

@Kaitlynyau

Also worth noting, the system light on our spotter is now solid red… We are wondering if this could be related/the cause of the issue, since we know that that can indicate SD card issues. What can we do to try and restore or reformat the SD card, and will that require wiping it and starting fresh?

Indeed, SD card issues can cause this. The SD card needs to be formatted FAT32. If you are comfortable using the USB serial console to communicate with the Spotter and Dev Kit, then you could connect to Spotter’s USB serial console and look for error messages. It is quite likely that Spotter will tell you more about its error condition.

A red System LED can also be caused by low battery, or other conditions.

[elliepeauroi]

Hi Tim! Thank you for the response and help. I’m working with Kaitlyn and I have some more information to share about our system. I’ll address your questions directly:

1. The application we tried is the .elf.dfu.bin file downloaded directly from the “Integrating an RS232 Serial Sensor” guide.
2. No, we have’nt had success with re-flashing it from any other .bin files found in the dev kit guides.
3. We’ve been using the spotter method, following the steps for “Updates over Bristlemouth” in the Mote Firmware Updates guide (although we are ordering the SDW debugger that will hopefully make things easier in the future)

All of this is really leading me to believe we have corrupted our SD card, which certainly could have happened from sloppy transfers of it from the spotter to sd card reader and back.
So I checked that it was in fact formatted FAT32 then I ran the first aid tool in the mac Disk Utility app. It said that it fixed a few errors but preserved all the data.
The spotter’s console doesn’t give any errors, but attempting to re-flash the mote again still fails with err:ffffffff. I find it suspicious because the spotter also isn’t outputting all of the same data as it did out of the box. While it used to give all kinds of info about power/pressure etc, the only info now is the initial message with notecard/cellular info and then a repeated FFT begin, FFT complete, sample analysis complete. I have no idea what this means, but for the sake of diagnosing I’ll paste the output of entering ‘error debug’ to the spotter console (it looks bad):

0
0
3
0
0
0
0
0
2
0
0
0
0
0
1
0
0
2
1
1
2
1

So… I feel like an issue with the spotter not the mote, and it could be the SD card since I can see how we could’ve messed that up. What do you think, do you agree? Or are we missing something. Is there a way to restore the SD card more than what I already tried?

Another quick note, we’re having another problem that may be tied up in this big issue but it’s puzzling so I want to mention it now. I probed the pins on the dev board with a DMM just to see if we are getting power, and we aren’t. The 3.3v is the only power pin that seems to be on, while the 5V and VOUT roll in with a fat 0V. I’m hoping thats a bug that will be fixed with one of the firmware updates we haven’t been able to install yet?

Thank you for the help, we really appreciate it!

[vsowa12]

Hey @Kaitlynyau and @elliepeauroi in the Spotters console could you please type “post” and respond with the output?

The 5V and VOUT can be enabled by “gpio set/clr <pin_name>” in the motes console or in the application code. (Some of the pins are reverse polarity so try both set and clr, you can check bm_protocol/src/lib/bristlefin/bristlefin.cpp for polarity)

The serial_payload_example will automatically enable VOUT, but will not enable 5V unless the application code is modified.

[Kaitlynyau]

Here is the result of “post”:

[vsowa12]

Thanks! So I think I know what happened here.

Context:
So here we can see “bridgeErrorState: SELF_TEST_ERR”. This “bridge” is a separate board with its own MCU in the Spotter ebox that acts as the Spotters Bristlemouth node. On the Bristlemouth network the bridge participates just like any other node and has its own unique node id. To make sure Spotter and the bridge can talk, the Spotter issues a self test command to the bridge and in this case it failed, hence the SELF_TEST_ERR. This causes the power to the bridge (and to bristlemouth) to be turned off.

My theory:
I think when updating the mote the bridge was actually the target of the update, thus the bridge had “mote” firmware running on it and is no longer able to talk to the Spotter, so when the Spotter is reset it powers the bridge down after the failed self test. This would explain 1. why the info on your mote didn’t change and 2. why the red system led is on. By chance do you remember the node id that was used in the command for the update? You can compare this node id to the one on the mote. If they do not match then it was likely the node id of the bridge.

There is a possibility that we can still talk to the bridge via USB. To test and see if we can still talk to the bridge via USB will require a few things: a micro-usb cable, a 5mm hex wrench, and a usb-c cable. If we can talk to the bridge this file https://github.com/bristlemouth/bm_protocol/releases/download/v0.4.0/bridge_v1_0-bridge.elf.unified.bin will be used to recover the bridge.

In order to access the micro-usb port to serial into the bridge you will need to do the following:

1. remove the ebox from the spotter hull
2. using the 5 mm hex wrench, unscrew the two hex bolts on the bottom of the ebox and remove the battery cover
3. unstrap and remove the battery, but leave it connected to the ebox. Underneath the battery there should be a micro-usb port.
4. plug in the micro-usb into the exposed port and the other end into your computer
5. plug in the usb-c into the spotter and the other end into your computer
6. turn on the spotter and serial in. Now here the Spotter has a reboot controller that will likely reset Spotter 1-2 times as it tries to recover the bridge, so wait 10-30 seconds and see if it resets and just reconnect the serial each time.
7. Once the Spotter is done resetting itself, type “gpio set 3v3_bridge_en” and you should get an “OK” response
8. Now, look for a bristlemouth serial device to connect to. If one shows up it is the bridge and you should be able to connect to it and perform a dfu update.
9. If the bridge shows up in your serial devices list, connect to it (you can do an “info” here to see what it is running). Issue the “bootloader” command. This will put the bridge in the STM32U5’s ROM bootloader.
10. In terminal use “dfu-util -l” to confirm that there is a device in dfu mode.
11. If there is a device in dfu mode then you can issue “dfu-util -d 0483:df11 -c 1 -i 0 -a 0 -s 0x8000000:leave -D bridge_v1_0-bridge.elf.unified.bin”
12. after it is completed try to serial back into the bridge and give it an “info” again to see if it is now running the bridge firmware again.
13. If all worked, reset the spotter and it should no longer have the self test error.

If you get to Step 8. and there is nothing showing up in the list of attached serial devices then if you are interested we can pull apart the ebox even more(this will require a T10 screwdriver) to access the boot pin on the bridge and manually put it into dfu mode. Or we can coordinate having the ebox sent back to Sofar to fix it for you (this is also an option if you are not comfortable with the above procedure). Don’t hesitate to ask for any further clarification!

[Kaitlynyau]

Hello again,
So we took to the advice you laid out for us but were not able to find any serial device connections when we got to step 8. Could you clarify step 8 from your last response? I’m not sure if we’re meant to try to connect to the bridge from a new terminal or through the existing connection to the spotter. With a USB connection from the bridge to my computer, the only device available for connection is the spotter, which does not seem promising. If we are meant to connect thru the spotter, what is the command to do so? Also is the “spotter ID” that can be found using “bm topo” the bridge ID that we want to connect to?
This may have been foolish, but we did quickly try to update the bridge in the same way we updated the mote, using “bridge dfu <name of the bridge .bin you provided> 12000 force”. It exited with Error -1.
We also noticed that the resets the spotter was going through were self-exiting stating that device was not configured. Ill attach screenshots of what our console currently looks like upon start up.

We’re not sure if these snippets of our terminal indicates the state of our system but we hope we can get to the bottom of this problem soon!
Another small issue we’ve noticed amongst all of the troubleshooting tactics we’ve tried is pulling up the “bm topo” command in the Dev Kit Serial Console only responds with the ID of the Dev Kit and not the spotter:

We are not sure if this specifically means anything but have just noticed that it is not behaving as it once did.

[vsowa12]

Hello!

Yes in step 8 you are trying to connect to the bridge from a new terminal. The bridge has its own serial CLI that can be used to update it.

The reboots from the Spotter are its way of attempting to recover a missing/broken peripheral, so yes, it is the Spotter that is ending the serial connection.

It sounds like the bridge is in a really poor state, and by that I mean it is stuck in its MCUBoot bootloader. I can explain more on how it got into this state if interested but I will go ahead and list the steps needed to recover it. For this you will need to open up the ebox even more and once again if you are uncomfortable with these steps we can coordinate getting the ebox back to Sofar where we will fix it for you and send it back.

For this you will need dfu-util on your computer and a T-10 screwdriver like this:

You can install dfu-util on a mac using the following command in the terminal.

brew install dfu-util

For linux check out the dfu-util page: https://dfu-util.sourceforge.net/

Steps:

1. unplug the battery from the bottom of the ebox
2. using the T-10 screwdriver, unscrew the 4 ebox screws and remove the PCBs from the ebox.
   
   
3. you should now have a PCB board stack that looks like this from the top:
   
   

and this from the bottom:

3. Plug the battery back into the ebox

4. connect the usb-micro to the bridge and the usb-c to the spotter. You should have something looking like this:
   

   
   
   
   

5. Orient the board stack such that you have access to these buttons on the bottom:
   

   
   
   Take note of the BOOT1 button to the right of the micro-usb and the RST1 button on the far left!

6. Turn on the spotter

7. Open two terminal windows. In one terminal, connect to the Spotter CLI. Once again wait until the spotter is done resetting itself, then issue “gpio set 3v3_bridge_en” to turn on power to the bridge

8. Press and hold the BOOT1 button, then without releasing BOOT1, tap on the RST1 button, then you can release BOOT1. This will put the bridge into its ROM bootloader and from there we can update it with dfu-util.

9. In the second terminal window check to make sure the bridge is accesible to dfu-util by issuing:

dfu-util -l

you should see something like:

dfu-util 0.11

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2021 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

Found DFU: [0483:df11] ver=0200, devnum=5, cfg=1, intf=0, path="0-1.4", alt=2, name="@OTP Memory   /0x0BFA0000/01*512 e", serial="208937834D30"
Found DFU: [0483:df11] ver=0200, devnum=5, cfg=1, intf=0, path="0-1.4", alt=1, name="@Option Bytes   /0x40022040/01*64 e", serial="208937834D30"
Found DFU: [0483:df11] ver=0200, devnum=5, cfg=1, intf=0, path="0-1.4", alt=0, name="@Internal Flash   /0x08000000/256*08Kg", serial="208937834D30"

If you dont see the last three lines then the bridge is not in the ROM bootloader and I would double check that 1. it is getting power and 2. the button sequence was pressed correctly.

9. If the bridge is present, then issue the following command to update the bridge (make sure the file path is pointing to where you have your bridge.bin file)

dfu-util -d 0483:df11 -c 1 -i 0 -a 0 -s 0x8000000:leave -D ~/Downloads/bridge_v4_0-bridge.elf.unified.bin

10. You should see the following after dfu-util is complete:

dfu-util 0.11

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2021 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

dfu-util: Warning: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release
Opening DFU capable USB device...
Device ID 0483:df11
Device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Interface #0 ...
Determining device status...
DFU state(2) = dfuIDLE, status(0) = No error condition is present
DFU mode device DFU version 011a
Device returned transfer size 1024
DfuSe interface name: "Internal Flash   "
Downloading element to address = 0x08000000, size = 250120
Erase   	[=========================] 100%       250120 bytes
Erase    done.
Download	[=========================] 100%       250120 bytes
Download done.
File downloaded successfully
Submitting leave request...
dfu-util: Error during download get_status

(don’t mind the error at the end)
11. You should then be able to see the following in your available serial ports:

---  3: /dev/cu.usbmodem000022DFB3741 'Bristlemouth'
---  4: /dev/cu.usbmodem000022DFB3743 'Bristlemouth'
---  5: /dev/cu.usbmodem1301 'Spotter'

12. You can connect to the 3: /dev/cu.usbmodem000022DFB3741 'Bristlemouth' device and type info and verify that it says it is a “bridge”
13. You can then toggle the on switch for spotter to reset it and reconnect to its CLI then check post to verify that bridgeErrorState is now “OK”.

[elliepeauroi]

That worked!!!
I am actually jumping for joy right now.

We were able to flash the mote (spotter method) with a fresh new firmware version and saw that the app successfully changed. Such relief.

We’ve ordered an SWD-USBC adaptor that we plan to use moving forward. At least if we break the bridge again, now we know how to fix it!

Thank you guys for sticking with us!

[vsowa12]

Woooohoooo! Congratulations! That is great to hear! I’m glad I was able help!

Thank you for pushing the boundaries and posting about it!

Good luck with your project!