SARA R410M loses USB connection to RPi

I am developing a smart agriculture application on a Raspberry Pi 3 B+ (Buster). For cellular comms I use a Nova SARA-R410M-02B modems with a Hologram SIM card.
Also part of the application is an XBee RF radio that receives sensor data from the field. Both devices use serial IO via the Pi USB ports.

My application is written in Ruby and uses the Serialport gem for serial communications. I set up a udev rule that creates a symlink called ttyNOVA for the Nova modem and ttyXBEE for the SBee radio:

SUBSYSTEM==“tty”, ATTRS{idVendor}==“05c6”, ATTRS{idProduct}==“90b2”, KERNEL==“ttyUSB*”, SYMLINK+=“ttyNOVA”, MODE=“0666”
SUBSYSTEM==“tty”, ATTRS{idVendor}==“0403”, ATTRS{idProduct}==“6015”, KERNEL==“ttyUSB*”, SYMLINK+=“ttyXBEE”, MODE=“0666”

When the Pi boots up everything is set up ok:

lrwxrwxrwx 1 root root 7 Mar 5 08:45 ttyNOVA -> ttyUSB2
lrwxrwxrwx 1 root root 7 Mar 5 08:45 ttyXBEE -> ttyUSB0

After opening the serial port for ttyNOVA the port functions normally with:

cell port signals = {“rts”=>1, “dtr”=>1, “cts”=>1, “dsr”=>0, “dcd”=>0, “ri”=>0} (Note that “cts”=>1)

After starting the application everything hums along until the serial port is suddenly lost. Might take a few minutes or many hours.
Grepping the syslog for entries with “usb” in them I find a pattern like this:

Mar 2 14:52:38 raspberrypi kernel: [ 131.566043] usb 1-1.3: USB disconnect, device number 8
Mar 2 14:52:38 raspberrypi kernel: [ 131.568025] qmi_wwan 1-1.3:1.3 wwan0: unregister ‘qmi_wwan’ usb-3f980000.usb-1.3, WWAN/QMI device
Mar 2 14:52:39 raspberrypi kernel: [ 132.632517] usb 1-1.3: new high-speed USB device number 9 using dwc_otg
Mar 2 14:52:39 raspberrypi kernel: [ 132.763246] usb 1-1.3: New USB device found, idVendor=05c6, idProduct=90b2, bcdDevice= 0.00
Mar 2 14:52:39 raspberrypi kernel: [ 132.763263] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Mar 2 14:52:39 raspberrypi kernel: [ 132.763273] usb 1-1.3: Product: QHSUSB__BULK
Mar 2 14:52:39 raspberrypi kernel: [ 132.763282] usb 1-1.3: Manufacturer: Qualcomm CDMA Technologies MSM
Mar 2 14:52:39 raspberrypi kernel: [ 132.763291] usb 1-1.3: SerialNumber: db4cad1d
Mar 2 14:52:39 raspberrypi kernel: [ 132.764775] usb 1-1.3: GSM modem (1-port) converter now attached to ttyUSB1
Mar 2 14:52:39 raspberrypi mtp-probe: checking bus 1, device 9: “/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.3”
Mar 2 14:52:39 raspberrypi mtp-probe: checking bus 1, device 9: “/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.3”

There is no other USB activty in the syslog around this time. Any idea what might be causing this? XBee port ‘interference’?

After this, the next time the Nova serial port is accessed (eg. to = serialport.read_timeout) an exception is thrown:

Input/output error - tcgetattr on port /dev/ttyNOVA

As far as I can tell, this error implies that the serial port no longer exists, or is in some way invalid.
This makes sense as ttyNOVA is now linked to a disconnected port (ttyUSB2) but reconnected to another port (ttyUSB1).

Currently, the application is designed to shutdown and restart at this point (sudo shutdown -r now).
Often, it comes up and there are two immediate observations.
First, after opening the ttyNOVA port we get:

cell port signals = {“rts”=>1, “dtr”=>1, “cts”=>0, “dsr”=>0, “dcd”=>0, “ri”=>0} (Note that “cts”=>1)

Note that “cts”=>0. Whenever this happens it indicates the serial port is not properly initialized.
No AT commands execute successfully. Bytes appear to be written but usually there is no valid response.
Sometimes there is a ‘response’ of sorts, but it is nonsense. For example, often I will see a byte pattern something like

1, 0, 0, 0, 48, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0, 0, 4, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0

This response has 48 bytes, but sometimes there are hundreds.

Second, before issuing the first AT command (ATE0) the input buffer is cleared. Invariably if there are characters in the buffer, they are always the same:

0x60 0x0a 0x00 0x9e 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xef 0xad 0x7e

followed by a ‘nil’. Anyone recognize this byte pattern (I don’t)?

Usually after a few restarts, things return to normal and communications are restored.

I should also note that I have three test systems running atm and only one is exhibiting this poor behaviour. Also, the problem modem has had the latest firmware upgrade installed (https://support.hologram.io/hc/en-us/articles/360035212594-Updating-the-Cat-M1-R410-Nova-s-Firmware).

I have the same going on with a Pi. I’m using the hologram python api and the R410m and have also updated firmware to latest. When the modem stops working, both the leds are out. I’ve made sure I’m using a substantial power supply.

The only pattern I think I’ve noted is that the problem happens much quickly if I’m not interacting with the modem (listening for inbound cloud data).

There are normally two ttyusb devices for the modem. After the failure, there is only one.

Yes, I’ve also noticed the problem with the LEDs. It seems particularly odd that the blue power LED would stop working. I use only the standard power blocks that come with the basic RPi3B+. This is a 2.5A supply which should be more than sufficient. The only external draw on the PS is a small cooling fan which only comes on rarely and does not appear to be connected with the issue.

Sometimes the problem takes many hours or even days before it occurs. The syslog indicates that, until the port is disconnected as shown in the original post, there is no other USB activity from any source. Also, the application is running basically unattended so there is no human interaction either.

I just ran another test and found that after a restart the app started up correctly: initialized the modem, connected to the network, read some modem information, etc - IOW, everything is working exactly as it should. The next thing the app tried to do was register MQTT agent information. This was immediately followed by a system disconnect (as indicated in the original post):

Mar 9 09:25:01 raspberrypi kernel: [ 127.302880] usb 1-1.4: new high-speed USB device number 9 using dwc_otg
Mar 9 09:25:01 raspberrypi kernel: [ 127.434059] usb 1-1.4: New USB device found, idVendor=05c6, idProduct=90b2
Mar 9 09:25:01 raspberrypi kernel: [ 127.434077] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Mar 9 09:25:01 raspberrypi kernel: [ 127.434088] usb 1-1.4: Product: QHSUSB__BULK
Mar 9 09:25:01 raspberrypi kernel: [ 127.434098] usb 1-1.4: Manufacturer: Qualcomm CDMA Technologies MSM
Mar 9 09:25:01 raspberrypi kernel: [ 127.434109] usb 1-1.4: SerialNumber: 8b17c662
Mar 9 09:25:01 raspberrypi kernel: [ 127.435691] option 1-1.4:1.0: GSM modem (1-port) converter detected
Mar 9 09:25:01 raspberrypi kernel: [ 127.436317] usb 1-1.4: GSM modem (1-port) converter now attached to ttyUSB1
Mar 9 09:25:01 raspberrypi mtp-probe: checking bus 1, device 9: “/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.4”
Mar 9 09:25:01 raspberrypi mtp-probe: bus: 1, device: 9 was not an MTP device

Again, at this time - about 2 minutes after a restart of the Pi - there was no user interaction and no RF communications from the XBee. Only the spontaneous disconnection and re-connection of the USB port to which the modem was attached, leading to an exception the next time the port was accessed (for the MQTT init).

Have you tried the “Troubleshooting” section on that firmware upgrade page? I’ve found that doing that refresh of the MNO Profile settings can help with a case like this.

That was something I planned to do because sometimes the modem does not make or maintain a connection with the carrier. However, the issue in this post is more about connecting and retaining a USB serial connection with the modem. Maybe the issues are connected? In any case, I can’t issue an AT command unless I can establish a connection to the modem.

Also, if I want to use the AT+UMNOPROF= command I need to know the MNO number of my carrier, Rogers (Rogers Communication Partnership). I have not been able to find it listed anywhere. Do you know what it might be?

Regards.

For whatever reason, refreshing the profile back and forth seems to help with the board itself powering off. My theory is that the firmware upgrade might not be restoring all files properly in all cases and doing this forces things to refresh.

You can just set the profile number back to whatever you’re already using if it’s working for you right now, or you can follow the instructions in the UMNOPROF and Bandmask section of this guide: https://support.hologram.io/hc/en-us/articles/360038810214-u-blox-SARA-R410

That will get you to a pretty general profile that should work on carriers that we support.

After switching to the 100 profile and restarting, I get an error on the first bandmask. The second one seems OK. I ended up putting the bandmask from profile zero in for the errored one and the connection to ATT popped right up. I’ll report back on the power issue.
I’m wondering if the power issue is the modem going into deep sleep mode, even though that appears to be disabled.

The shutdown problem happened within a few hours of being up on profile 100. Maybe a hardware issue? How do I swap out the modem?

Not sure if I saw that above, but what kind of power supply are you using on the Pi?

If you want to do an exchange, send us an email at success@hologram.io and someone can help you out.

I’ve tried both a 3A PS that came with the Pi as well as a big HP laptop supply. The modem shuts off after a random period of time even when plugged into a powered USB hub, without the pi connected at all.

Reuben,
I haven’t had any success getting a response from the success team. Could you give them a nudge for me please?
Jon

Yeah I’ll have them look out for your message