SARA R410M loses USB connection to RPi

@Reuben: Do you have enough details to bring this issue to ublox?

@genosensor Trying to understand the underlying problem here. You’re saying that when using an R410 on a Pi, you can’t maintain a PPP session for more than 10 seconds without it resetting back to the debug port?

Can you describe how you’re powering the Pi?

Are both of you (@Sven @genosensor) running the latest firmware? Did this start happening after the upgrade and have you tried the troubleshooting trick described above?

Regarding the missing caps, those are only needed for the U201 Nova so we’d recommend not populating those pads on the R410.

Regarding your note 4, are you saying that adding those filter caps on the USB lines fixes the problem entirely?

Yes, running the latest firmware and I have also tried the troubleshooting trick. No change.

I do not see the issue as frequently as genosensor, but it does happen enough to make the solution not usable (e.g. sometimes no issue for 1 to 2 weeks, but sometimes two crashes within 10 minutes).

To answer the power question: I tried both a 2.5A power supply with the USB port and now use a 3A power supply directly to the 5V pins on the RP header. There is no voltage drop on the modem when receiving or sending. That was the first thing it measured. :slight_smile:

EDIT for completeness. Here is what is logged in syslog when this issue happens.
My code is only querying for SMS message when this happens.

Apr 12 21:07:35 hangarcontroller kernel: [ 122.752875] ERROR::dwc_otg_hcd_urb_enqueue:501: Not connected
Apr 12 21:07:35 hangarcontroller kernel: [ 122.752875]
Apr 12 21:07:35 hangarcontroller kernel: [ 122.753017] ERROR::dwc_otg_hcd_urb_enqueue:501: Not connected
Apr 12 21:07:35 hangarcontroller kernel: [ 122.753017]
Apr 12 21:07:35 hangarcontroller kernel: [ 122.753050] ERROR::dwc_otg_hcd_urb_enqueue:501: Not connected
Apr 12 21:07:35 hangarcontroller kernel: [ 122.753050]
Apr 12 21:07:35 hangarcontroller kernel: [ 122.753069] ERROR::dwc_otg_hcd_urb_enqueue:501: Not connected
Apr 12 21:07:35 hangarcontroller kernel: [ 122.753069]
Apr 12 21:07:35 hangarcontroller kernel: [ 122.797063] usb 1-1: USB disconnect, device number 2
Apr 12 21:07:35 hangarcontroller kernel: [ 122.797783] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
Apr 12 21:07:35 hangarcontroller kernel: [ 122.797883] option 1-1:1.0: device disconnected
Apr 12 21:07:35 hangarcontroller kernel: [ 122.799292] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
Apr 12 21:07:35 hangarcontroller kernel: [ 122.799366] option 1-1:1.2: device disconnected
Apr 12 21:07:35 hangarcontroller kernel: [ 122.800595] qmi_wwan 1-1:1.3 wwan0: unregister ‘qmi_wwan’ usb-3f980000.usb-1, WWAN/QMI device
Apr 12 21:07:35 hangarcontroller avahi-daemon[328]: Interface wwan0.IPv6 no longer relevant for mDNS.
Apr 12 21:07:35 hangarcontroller avahi-daemon[328]: Leaving mDNS multicast group on interface wwan0.IPv6 with address fe80::2e8c:faaf:7eb4:acb0.
Apr 12 21:07:35 hangarcontroller dhcpcd[417]: wwan0: carrier lost
Apr 12 21:07:35 hangarcontroller avahi-daemon[328]: Interface wwan0.IPv4 no longer relevant for mDNS.
Apr 12 21:07:35 hangarcontroller avahi-daemon[328]: Leaving mDNS multicast group on interface wwan0.IPv4 with address 169.254.181.0.
Apr 12 21:07:35 hangarcontroller avahi-daemon[328]: Withdrawing address record for fe80::2e8c:faaf:7eb4:acb0 on wwan0.
Apr 12 21:07:35 hangarcontroller avahi-daemon[328]: Withdrawing address record for 169.254.181.0 on wwan0.
Apr 12 21:07:35 hangarcontroller dhcpcd[417]: wwan0: deleting address fe80::2e8c:faaf:7eb4:acb0
Apr 12 21:07:35 hangarcontroller dhcpcd[417]: wwan0: deleting route to 169.254.0.0/16
Apr 12 21:07:35 hangarcontroller dhcpcd[417]: wwan0: deleting default route
Apr 12 21:07:35 hangarcontroller dhcpcd[417]: wwan0: removing interface
Apr 12 21:07:36 hangarcontroller kernel: [ 124.007013] Indeed it is in host mode hprt0 = 00021501
Apr 12 21:07:36 hangarcontroller kernel: [ 124.216874] usb 1-1: new high-speed USB device number 3 using dwc_otg
Apr 12 21:07:36 hangarcontroller kernel: [ 124.216968] Indeed it is in host mode hprt0 = 00001101
Apr 12 21:07:36 hangarcontroller kernel: [ 124.457378] usb 1-1: New USB device found, idVendor=05c6, idProduct=90b2, bcdDevice= 0.00
Apr 12 21:07:36 hangarcontroller kernel: [ 124.457384] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Apr 12 21:07:36 hangarcontroller kernel: [ 124.457389] usb 1-1: Product: QHSUSB__BULK
Apr 12 21:07:36 hangarcontroller kernel: [ 124.457392] usb 1-1: Manufacturer: Qualcomm CDMA Technologies MSM
Apr 12 21:07:36 hangarcontroller kernel: [ 124.457396] usb 1-1: SerialNumber: 1ae5f4e6
Apr 12 21:07:36 hangarcontroller kernel: [ 124.457894] option 1-1:1.0: GSM modem (1-port) converter detected
Apr 12 21:07:36 hangarcontroller kernel: [ 124.458072] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB0
Apr 12 21:07:36 hangarcontroller mtp-probe: checking bus 1, device 3: “/sys/devices/platform/soc/3f980000.usb/usb1/1-1”
Apr 12 21:07:36 hangarcontroller mtp-probe: bus: 1, device: 3 was not an MTP device
Apr 12 21:07:37 hangarcontroller mtp-probe: checking bus 1, device 3: “/sys/devices/platform/soc/3f980000.usb/usb1/1-1”
Apr 12 21:07:37 hangarcontroller mtp-probe: bus: 1, device: 3 was not an MTP device

@Reuben We are not using a Pi of any sort. In 2015 we designed our own low-power ARM5 Linux CPU PC104 board. This is why I didn’t raise this issue until someone else reported it on more mainstream hardware – like the ubiquitous Pi.

I’ve tried powering our board various ways. Like @Sven, I don’t see any voltage dips below 4.9V.

Thanks for the info on the large bypass caps. That makes sense, as the U201 draws higher peak currents. Adding the big bypass caps certainly wouldn’t hurt anything. In my case, it made no difference.

Regarding note 4:
Adding the pair of small 47pf caps on the USB data lines allows us to plug the Nova modem directly into our board’s USB plug. Without those caps, there must be a 6ft or longer USB extension cable between the Nova and our board or a big ferrite choke. This problem is entirely separate from the firmware crashing issue @Sven and I both observe. I suspect the 47pf caps are only needed with some CPU boards, not others. However, I’ve never had problems with plugging any other device directly into our CPU’s USB port.
@Sven, have you been running the Nova plugged directly into the Pi, without any cabling or hubs between the two?

@Reuben We are running the .08 firmware.
If I take care to start the PPP session within 4 or 5 seconds of the R410 enumerating on the USB, they typically hold for hours before the R410 resets back the the debug port. At which point, our system now automatically cycles the USB power to automatically recover the connection. This is not good, but acceptable for us.
However…
If I wait to start the PPP session 30 seconds after the R410 enumerates on the USB, that session typically fails within less than two minutes, often with the R410 resetting back to its debug port configuration.

If you or @Reuben can provide a contact at uBlox, I would try to work with them. So far, I’ve had little luck reaching anyone competent there.
Also, our CPU board is a low-volume, custom design, which made me worry that there was some odd interaction between it and the R410 causing these crashes.

Directly into the PI.

The crashing R410 was first observed with the .06 firmware.
I managed to get the .08 firmware from uBlox last summer in the hope it would fix this.

You might try putting a long USB extension between the two. Report if that changes anything. It probably won’t in your case, but it did in mine, before I added the 47pf caps in the Nova board.

A way to solve or better bandage the problem is to wire the R410M reset pin to a GPIO.
image

The indicated pin needs to be pulled low for 10s to reset the modem. I tested this on the R410M.
Careful when soldering a cable onto the pad. Easy to rip the pad of if pulled on (tested that as well… :frowning: )

We’ve already run into our share of flakey modems. Cycling power works to reset all of them.
Our instruments are deployed underwater with a 30m cable to a “surface expression” (i.e. buoy) that houses the radio. There’s no programmable computer in the buoy. An ICRON 1850 extends the USB bus.
I wish we could simply tack on GPIOs :slight_smile:
Is that what you intend to do?
Have you verified that a 10s low on this reset line will recover the R410 from this “debugging” state?

I have not since I was not yet able to trigger that state.
Something we should get clarity on from hologram/ublox.

@davidm and I are seeing the modem reenumerate with a product string of QHSUSB__BULK , but you just see the modem hang up without reenumerating on the bus? Is this correct?
Have you looked at the output of the dmesg command just after one of these modem lockups? If so, could you please post the last 40 or so lines?

@davidm @Sven
What underlying service are you using?
I’m on AT&T.
I’ve been noticing that connections are stable during the night and typically fail at about 9:30AM. I’m starting to suspect that there might be an interaction between the these failures and the network congestion. Is that consistent with your experience?

I am on ATT.

Also agree that there is likely an impact from the tower the modem is connected to.
I see zero error in on location and errors in another location. Swapping the devices does not change the error location.

Error case is also that the modem goes into bulk recovery. Posted the logs earlier.

Just installed a SARA-U201 to replace the R410 in that location.
Will see how that goes.

Not happy about it, but looks like the R410 needs some more time to mature. There is clearly a firmware bug in it that needs work from ublox.

@Sven if you have a board that consistently has trouble with a normal Raspberry Pi, we’d like to swap it out. The new boards have the updated firmware preinstalled so it would be interesting to see your results with that and getting your board back will help us to debug with ublox. Can you reach out to support@hologram.io and someone can help you?

Just sent email to the support.

The modem works fine in one location, but has error in another location.

Is there firmware version newer than 05.08?

No, it’s the same one you can upgrade to. It’s just preinstalled