Nova firmware update fails

Hardware (and firmware) Nova
1

Hi,

I have a bunch of Nova devices with module SARA R410M-02B. They are all running firmware version:

L0.0.00.00.05.06 [Feb 03 2018 13:00:41]

I want to update the firmware as describe elsewhere on this forum. I followed these instructions,

hologram-tools/novaupdate at master · hologram-io/hologram-tools · GitHub

for installing the updater. I performed the update,

sudo python nova410update.py

and it worked! Then I attempted to update a second and third device and the updates failed. Here’s what happened:

$ sudo python nova410update.py

You are about to update the UBlox firmware on your Nova R410 device.
Once you continue past this point do not power off the modem
or stop this utility before it has completed.
Note that this update may change some behavior of the modem
related to profile selection and LED blinking. See hologram
documentation for more information.
The update may take up to 25 minutes to apply
Do you wish to continue? (y/n) y
Confirming modem type
Checking current modem version
Got version L0.0.00.00.05.06,A.02.00
Sending file /home/matt/spotter/hologram/nova/hologram-tools/novaupdate/fw/0200/L0506A0200-to-L0508A0204/0bb_stg1_pkg1-0m_L56A0200_to_L58A0204.bin
Writing file to serial port
ERROR: Failed to send file via xmodem

I tried several more times with the similar, but not identical results. Each attempt failed, but sometimes it tried sending 3 files, sometimes 2 files, and sometimes only 1 file.

Any suggestions?

Thanks,
Matt

2

Hello,

I was hoping for a response from one of the Hologram folks responsible for the Nova. I am experiencing some connectivity problems with the earlier firmware which seem to be solved with the newer firmware.

The devices with the older firmware are deployed in Mississippi while my office is in Virginia. I will have to visit the devices and update them on-site. That’s not going to be fun, but that’s my reality. Of course, I can’t update them–even in person–if the update software won’t perform the update. I’m willing to take all necessary steps, but I really do need your help on this. PLEASE RESPOND.

3

Hi, we haven’t seen that problem before. The error seems to indicate that it’s not even able to send the file so maybe the serial port is getting tied up. Are you using a Raspberry Pi to run the update? Which model of pi? Have you tried with a USB hub or anything like that? When the update fails are the LEDs turned on or off on the modem.
All modems are R410?
One thing to try is rebooting the pi between each board.

4

Hi Reuben,

Thanks for the reply. I’m using an HP notebook computer running Ubuntu 16.04. I’ve turned off the ModemManager service as it likes to manager the USB ports. I have installed the Hologram Python CLI package for controlling the Nova. Since I am able to use that package to interact with the Nova, it seems python is able to interact with the Nova.

I’m not using a USB hub. I simply plug the Nova into a USB port on the computer.

All attempts have been to update Nova R410 devices.

Before attempting the update I plug the Nova into a USB port. After a few seconds the blue LED lights. After another 10 seconds or so the red LED lights. Next, I run the installer. It checks the firmware version and reports it as:

L0.0.00.00.05.06,A.02.00

After about 10 seconds it displays:

sending file …/home/hologram/hologram-tools/novaupdate/…

After 20 more seconds it displays:

Writing fie to serial port

After 10 more seconds it displays:

Waiting for stage1 return code
Waiting for modem

At this point the LEDs go off. The blue LED lights again after about 5-10 seconds, and the red LED comes on about 10 seconds after the bue. After somewhere between 2 and 40 more seconds it displays:

Still waiting for modem to finish install. Do not unplug

Then it may pause for a while or not before displaying:

File failed. Trying next one in set’
sending file …/home/hologram/hologram-tools/novaupdate/…

It goes through this 1, 2, 3, or 4 times and ends with:

ERROR: Failed to send file via xmodem

Sometimes it fails with:

ERROR: (‘Unknown flag’, ‘55506b67’)

Just for the heck if it, I installed the Hologram SDK on an Orange Pi (similar to Raspberry Pi) running Linux and the fiirmware update softwa
re. I get identical results.

Any suggestions?

5

Yeah the reason we try to have people do it on a pi is because of things like the modem manager that can tie up the port. Sounds like you might be accounting for that though.

The updater should be dropping a log file in the directory in which it runs. Can you send that in a private message or via email to success@hologram.io

6

Would like to add that I am seeing exactly the same issue with same symptoms. Have tried on a Pi4 and on an ubuntu VM with the same errors on both. Pi4 is running raspbian buster.

2019-10-28 19:13:56,372 - WARNING - Writing file to serial port
2019-10-28 19:13:56,372 - DEBUG - Begin start sequence, packet_size=128
2019-10-28 19:14:17,398 - INFO - send error: error_count reached 20, aborting.
2019-10-28 19:14:17,399 - ERROR - ERROR: Failed to send file via xmodem
7

Just for the heck of it, can you guys try editing line 207 in the updater script to say retry=25 instead of retry=20

@mbrenner I have your logs. You’re actually seeing two different errors. We’ve reached out to ublox about one of them and hope to hear back soon.

8

Made the retry change and am still unable to update the firmware. I will send the full log to you.

Confirming modem type
Checking current modem version
Got version L0.0.00.00.05.06,A.02.01
Sending file /home/pi/nova/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/0bb_stg1_pkg1_0m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
Still waiting for modem to finish install. Do not unplug
File failed. Trying next one in set
Sending file /home/pi/nova/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/0bb_stg1_pkg2_4m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
Still waiting for modem to finish install. Do not unplug
File failed. Trying next one in set
Sending file /home/pi/nova/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/0bb_stg1_pkg3_8m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
Still waiting for modem to finish install. Do not unplug
File failed. Trying next one in set
Sending file /home/pi/nova/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/1bb_stg1_pkg1_0m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
Still waiting for modem to finish install. Do not unplug
Package set failed. Trying next one
Sending file /home/pi/nova/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/2bb_stg1_pkg1_0m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
Still waiting for modem to finish install. Do not unplug
Package set failed. Trying next one
ERROR: Was unable to install any update package successfully
9

Reuben,

I am having the exact same problem. I’m very interested in what the resolution is?

Thanks!

-DJ

10

Seems like there’s a couple different issues popping up here. Update is working fine for most people, but might be some subset of boards that require a different kind of package. We’re checking with ublox to see if they can provide more information.

11

Update here: We’ve been able to reproduce on one board in our lab and are passing debug information along to ublox to try to determine the cause. Hoping to have more information next week.

1 Like
12

Any update?

13

We’re still working with ublox on diagnosing this. Seems like the code coming back might indicate the module is in a weird state. We could use a couple more boards with the problem to help dig into this more. Could you reach out to support@hologram.io if you’re willing to help?

14

Reuben,

I’m willing to help but I’m also incredibly frustrated!

I just opened two brand new modems. Both of them do NOT work out of the box and need an update. The first one updated fine. The second one got this error:

pi@raspberrypi:~/hologram-tools/novaupdate $ sudo python nova410update.py

You are about to update the UBlox firmware on your Nova R410 device.
Once you continue past this point do not power off the modem
or stop this utility before it has completed.
Note that this update may change some behavior of the modem
related to profile selection and LED blinking. See hologram
documentation for more information.
The update may take up to 25 minutes to apply
Do you wish to continue? (y/n) y
Confirming modem type
Checking current modem version
Got version L0.0.00.00.05.06,A.02.01
Sending file /home/pi/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/0bb_stg1_pkg1_0m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
Still waiting for modem to finish install. Do not unplug
Sending file /home/pi/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/0bb_stg2_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for install to complete and modem to reconnect
This could take 20 minutes. Do not unplug the modem
Waiting for modem
Got version L0.0.00.00.05.06,A.02.01
ERROR: (‘Got unexpected modem version’, ‘01’)

It’s now completely bricked. It won’t power on and I do not see a serial port in linux when it is plugged in or any lights on the unit… What do I do now? Do I need a replacement and how can I get a replacement modem from Ublox / Hologram?

Is the Hologram Nova R410 ready for production environments? I’m worried it is not.

How can you help?

One last item. I cannot turn off the NB-IOT modem after the firmware upgrade “AT+URAT=7” yields: “+CME ERROR: Operation not supported”

How can I overcome this? Otherwise the modem takes an HOUR to connect to the network.

Lots of problems today.

Thanks,

-DJ

15

Let’s start with the last one first: The new firmware version changes how the URAT command works depending on which MNO profile is in use. See Appendix B.5 in the UBlox AT command manual for more information on that.

For your main issue, that’s something we haven’t seen before. It looks like it might have been a race condition in the updater where it checked the version before it was finished installing and thought it had failed to update when it hadn’t tried to run yet.

It’s possible that its now corrupted, and I think we can swap that out for you, but first, let’s make sure it’s totally bricked. Can you plug the unit back into the Pi and leave it for 25 minutes? That should give it enough time to finish the install if there’s any chance of recovery. If it worked, then you should see the serial ports again. May need to manually program the LEDs but I can post the commands for that if you can get to the serial port.

16

Hi Reuben,

I’ve been away from these dev modems for a spell - had to get a prototype out the door. I’m back now. So the one modem is definitely dead - won’t show any serial port and no lights. How do I get it replaced? Also, a second one powers on, lights up but will not take the firmware update. So strange. I have one more dev model in a box that I will try tomorrow - see if I can update that. I will report back.

Also, I’ve read the B.5 appendix. How do I turn off NBIOT? “at+urat=LTE CAT M1”. Maybe I’m reading the manual wrong. What do you think?

Cheers!

17

Can you reach out to support@hologram.io
There’s a log file for the updater that shows up in the directory you ran it in. Please attach that to the email.

18

To turn off NB you can pick a profile that either has it turned off already or where that setting is unlocked. That’s what the table in the appendix will show you. When you email support they can make some recommendations there for you

19

Hi all,

I’m having problems to connect my Nova via NB IoT. I opened a thread about that a few days back (https://community.hologram.io/t/nb-iot-usa/3434).
As I got no reaction there, I thought I might give the new Firmware Version a try. Unfortunately I also ran into Problems with the updater.
The output of nova410update.py is the following:

You are about to update the UBlox firmware on your Nova R410 device.
Once you continue past this point do not power off the modem
or stop this utility before it has completed.
Note that this update may change some behavior of the modem
related to profile selection and LED blinking. See hologram
documentation for more information.
The update may take up to 25 minutes to apply
Do you wish to continue? (y/n) y
Confirming modem type
Checking current modem version
Got version L0.0.00.00.05.06,A.02.01
Sending file /home/smartag/Downloads/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/0bb_stg1_pkg1_0m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
Still waiting for modem to finish install. Do not unplug
File failed. Trying next one in set
Sending file /home/smartag/Downloads/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/0bb_stg1_pkg2_4m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
Still waiting for modem to finish install. Do not unplug
File failed. Trying next one in set
Sending file /home/smartag/Downloads/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/0bb_stg1_pkg3_8m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
Still waiting for modem to finish install. Do not unplug
File failed. Trying next one in set
Sending file /home/smartag/Downloads/hologram-tools/novaupdate/fw/0201/L0506A0201-to-L0508A0204/1bb_stg1_pkg1_0m_L56A0201_to_L58A0204.bin
Writing file to serial port
Waiting for stage1 return code
Waiting for modem
ERROR: (‘Unknown flag’, ‘55506b67’)

Maybe sb hit the same error flag and has an Idea what I can try?
Any help is appreciated.

Best regards
Dominik

20

UBlox is still looking into that specific error code so unfortunately can’t provide any more info at this time. Feel free to reach out to us at support@hologram.io and we can try to swap that out for you. I’ll take a look at your other thread soon too.