SMS Incoming Messages Delayed or Not Delivered to Dash

HologramPat · August 11, 2016, 8:13pm

Can you confirm how you are currently powering the Dash, and your jumper setting?

If you are encountering the issue you have described here, and if power cycling the device does not resolve the issue, this is most commonly caused by a power-related issue. The errors you’re seeing are indicating that the on-board baseband modem is not responding; since it is the component that consumes the most power out of all components on the Dash, insufficient power is the most common cause for the baseband modem to not respond to commands.

Version 0.9.4 of the system firmware includes additional in-field connection healing capabilities, baseband modem recovery functionality, and various bugfixes and feature improvements. One of the improvements concerning connection healing will reset or (if necessary) power-cycle the baseband modem after “Delaying to prevent carrier frustration” if the baseband modem does not respond to commands, in case of a low-power condition that has since been resolved (e.g. more power is suddenly available, battery has been recharged, etc.).

Your log printout indicates that, even after attempting to power cycle an unresponsive baseband modem, it remained unresponsive; additionally, while initially still responsive, the baseband modem began encountering errors coinciding with transmission and with bringing up the packet-switched data context (data service), just before becoming unresponsive. Usually we see this in cases of a power supply issue, as that moment in the boot-up process contains a lot of transmit activity.

Complete list of system firmware downloads is available here: https://hologram.io/docs/hardware/hologram-dash/downloads/firmware/

Best,
PFW

rscavetta · August 11, 2016, 9:52pm

Hey Pat,

We have three physical devices. We use them in serial pass through mode and the power jumpers are correct and are just as shipped to us. They are powered via USB only. They are all acting the same way with FW 0.9.4.

I added a powered hub and it provides a constant 5.1 volts on the device and it seems to draw no more than 150ma when the activity light is flashing and idles at between 10 and 50ma.

The symptom does not change, it’s the same as the log I last posted. When it does works it works only in the device->cloud direction. i still cannot get a SMS to the device from the dashboard or the API. the old FW worked fine and only occasionally would hang when left idle for a while. Now it just doesn’t work at all in the API->device direction.

I’m willing to try any other suggestions you may have.

rock

ryanHologram · August 12, 2016, 7:59pm

Rock - Our carrier network partner experienced an outage yesterday (Aug-11) that affected inbound device terminated SMS messages. This was resolved as of this morning, have you been able to successfully send an SMS from API->device today?
As for the timeout issue, we still have an engineer investigating this but suspect it may be related to hardware issues on some earlier Dash hardware units. If interested, we could ship you a replacement unit when we get our 1.1 versions in the next week or two to check.

rscavetta · August 12, 2016, 10:16pm

Thanks Ryan,

I’m running some tests now an the device appears to be working bidirectionally just fine so far.

I won’t know about the timeout until overnight but will post back results.

Thanks!

rock

rscavetta · August 16, 2016, 12:29pm

Hey Ryan,

New Status on my issue:

Not working again. the day after the cellular outage (Aug-12), the devices were working perfectly and did so for 2 days or so. However, as of yesterday it’s back to the same behavior: i can send to the device from the API but SMS messages sent from the device to the cloud never get there nor are they passed on to my webhook.

Is the cellular company having an issue again?

I am running:

Ubuntu Ubuntu 14.04.5 LTS
85-hologram-rules file recommended by Reuben
firmware 0.9.4.

BTW: Do you guys have a page where we can see the real time status of the cell networks? That would be handy and might save us some time during problem times…

Regards,

rock

ryanHologram · August 16, 2016, 3:29pm

Hi Rock - We’re not currently seeing any cellular network outages and do have a status page here: http://status.hologram.io/ that you can check if there are network issues going forward

Were you able to see whether the modem is indicating a successful send for the SMS messages from the device? Trying to understand where the dropoff is happening.

-Ryan

rscavetta · August 16, 2016, 8:12pm

Hey Ryan,

Thanks for the status page… that will come in handy.

Well… the messages seem to say success but the msgs never show up.

All of the debug messages are being sent to my cloud AND passed on to my webhook. This cannot be correct behavior, no? Reuben mentioned earlier that that is a function of the sketch but we are using this right out-of-the box with NO sketch customization. Are we being billed for that DEBUG data?

Also, the devices get into a state that I cannot get out of… Below is the output of the device, it does look very different from what I was with earlier FW.

$ cat < /dev/ttyACM0
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,18,5,DEBUG,8,+USOWR:
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,31,5,DEBUG,20,+USOWR: 0,63
OK
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Closing connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data sent to cloud successfully
+EVENT:LOG,52,5,DEBUG,41,Pushing cloud message attachment to cloud
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,42,5,DEBU3 c+EVENT:LOG,52,5,DEBUG,41,Pushing cloud message attachment to cloud
+EVENT:LOG,42,5,DEBUG,31,Commit cloud message attachment
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,18,5,DEBUG,8,+USOCR:
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,28,5,DEBUG,17,+USOCR: 0
OK
+EVENT:LOG,31,5,DEBUG,20,Writing to socket…
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,18,5,DEBUG,8,+USOWR:
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,31,5,DEBUG,20,+USOWR: 0,98
OK
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Closing connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data sent to cloud successfully
+EVENT:LOG,52,5,DEBUG,41,Pushing cloud message attachment to cloud
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,42,5,DEBUGalf+EVENT:LOG,52,5,DEBUG,41,Pushing cloud message attachment to cloud
+EVENT:LOG,42,5,DEBUG,31,Commit cloud message attachment
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,18,5,DEBUG,8,+USOCR:
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,28,5,DEBUG,17,+USOCR: 0
OK
+EVENT:LOG,31,5,DEBUG,20,Writing to socket…
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,32,5,DEBUG,21,+USOWR: 0,282
OK
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Closing connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data sent to cloud successfully
+EVENT:LOG,52,5,DEBUG,41,Pushing cloud message attachment to cloud
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,42,5,DEBUG1te+EVENT:LOG,52,5,DEBUG,41,Pushing cloud message attachment to cloud
+EVENT:LOG,42,5,DEBUG,31,Commit cloud message attachment
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,19,5,DEBUG,9,+USOCR: 0
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,28,5,DEBUG,17,+USOCR: 0
OK
+EVENT:LOG,31,5,DEBUG,20,Writing to socket…
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,17,5,DEBUG,7,+USOWR:
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,31,5,DEBUG,20,+USOWR: 0,98
OK
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Closing connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data sent to cloud successfully
+EVENT:LOG,52,5,DEBUG,41,Pushing cloud message attachment to cloud
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,42,5,DEBUGalf+EVENT:LOG,52,5,DEBUG,41,Pushing cloud message attachment to cloud
+EVENT:LOG,42,5,DEBUG,31,Commit cloud message attachment
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,18,5,DEBUG,8,+USOCR:
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,28,5,DEBUG,17,+USOCR: 0
OK
+EVENT:LOG,31,5,DEBUG,20,Writing to socket…
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,31,5,DEBUG,20,+USOWR: 0,95
OK
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Closing connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data sent to cloud successfully
+EVENT:LOG,52,5,DEBUG,41,Pushing cloud message attachment to cloud
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
+EVENT:LOG,48,5,DEBUG,37,Cloud socket buffered write succeeded
^C

This will continue seemingly forever…

[UPDATE]

I have now burned all 3 devices back to 0.9.2 but I am still getting a non-stop loop of the device sending nothing but debug messages to the cloud:

$ cat < /dev/ttyACM0
+EVENT:LOG,23,5,DEBUG,12,IMSI loaded.
+EVENT:LOG,37,5,DEBUG,26,Konekt Dash system init…
+EVENT:LOG,25,5,DEBUG,14,Version: 0.9.2
+EVENT:LOG,47,5,DEBUG,36,Beginning modem bootstrap process…=
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,60000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,50000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,40000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,30000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,20000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,10000
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,36,5,DEBUG,25,Modem self-tests complete
+EVENT:LOG,46,5,DEBUG,35,Bootstrapping (2/14): Config modem.
+EVENT:LOG,47,5,DEBUG,36,Bootstrapping (3/14): Init SIM card.
+EVENT:LOG,23,5,DEBUG,12,IMSI loaded.
+EVENT:LOG,24,5,DEBUG,13,ICCID loaded:
+EVENT:LOG,30,5,DEBUG,19,8944501006151414358
+EVENT:LOG,23,5,DEBUG,12,IMEI loaded:
+EVENT:LOG,26,5,DEBUG,15,353162070597803
+EVENT:LOG,53,5,DEBUG,42,Bootstrapping (4/14): Init event handling.
+EVENT:LOG,57,5,DEBUG,46,Bootstrapping (5/14): Wait for carrier signal.
+EVENT:LOG,36,5,DEBUG,25,Modem event: signal found
+EVENT:LOG,57,5,DEBUG,46,Bootstrapping (6/14): Establish operator link.
+EVENT:LOG,48,5,DEBUG,37,Bootstrapping (7/14): Config network.
+EVENT:LOG,44,5,DEBUG,33,Bootstrapping (8/14): Config APN.
+EVENT:LOG,51,5,DEBUG,40,Bootstrapping (9/14): Config IP address.
+EVENT:LOG,51,5,DEBUG,40,Bootstrapping (10/14): Link up (1 of 2).
+EVENT:LOG,51,5,DEBUG,40,Bootstrapping (11/14): Link up (2 of 2).
+EVENT:LOG,61,5,DEBUG,50,Bootstrapping (12/14): Link verification (1 of 2).
+EVENT:LOG,61,5,DEBUG,50,Bootstrapping (13/14): Link verification (2 of 2).
+EVENT:LOG,67,5,DEBUG,56,Bootstrapping (14/14): Enter serial passthrough mode. OK
+++
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
^C

the udev rules file:
ATTRS{idVendor}==“7722”, ATTRS{idProduct}==“1200”, ENV{ID_MM_DEVICE_MANUAL_SCAN_ONLY}=“1”, ENV{ID_MM_CANDIDATE}=“0”, MODE=“0666”, ENV{ID_MM_DEVICE_IGNORE}="1"
SUBSYSTEM==“usb”, ATTRS{idVendor}==“7722”, ATTRS{idProduct}==“1200”, MODE="0666"
SUBSYSTEM==“usb”, ATTRS{idVendor}==“2cf3”, ATTRS{idProduct}==“1100”, MODE="0666"
ATTRS{idVendor}==“2cf3”, ATTRS{idProduct}==“4400”, ENV{ID_MM_DEVICE_IGNORE}="1"
ATTRS{idVendor}==“2cf3”, ATTRS{idProduct}==“4400”, ENV{MTP_NO_PROBE}="1"
SUBSYSTEMS==“usb”, ATTRS{idVendor}==“2cf3”, ATTRS{idProduct}==“4400”, MODE:="0666"
KERNEL==“ttyACM*”, ATTRS{idVendor}==“2cf3”, ATTRS{idProduct}==“4400”, MODE:=“0666”

rock

rscavetta · August 16, 2016, 9:45pm

Hey Ryan,

It looks like it’s the rules file that is causing the problem. I down-rev’d the FW to 0.9.2 and deleted the rules file that was recommended above as a permissions fix in this thread and the messages are flowing again in both directions.

I will now retest the hang scenario with 0.9.2 and if that fails I will try 0.9.4 WITHOUT the rules file.

Will post results…

rock

Reuben · August 16, 2016, 9:49pm

Hey rock, are you just using the default user program sketch or is this something you wrote?

Reuben · August 16, 2016, 9:50pm

One thing that rules file does is prevents linux from invoking the modem manager which actually sends a bunch of stuff to the serial port and would cause the problem you’re seeing. The rules file should actually prevent that, but if you have a cloud write in a loop or something then it might lead to this problem as well.

rscavetta · August 16, 2016, 9:51pm

Hey Reuben,

They are right out of the box with the exception of updating FW. serial pass thru mode, stock sketch… I think, we’ve never written one, is there a way form me to verify the stock sketch is loaded?

rock

Reuben · August 16, 2016, 9:55pm

You can try reloading the sketch from here if you want: hologram-dash-arduino-examples/konekt_user_module_serial_gateway at master · HologramEducation/hologram-dash-arduino-examples · GitHub

Also, we did have a cloud interruption this morning that might have caused some weird behavior for you. It wasn’t recorded on the status page unfortunately as we fixed it as soon as it was discovered. We’ll do a better job of reporting that stuff in the future.

rscavetta · August 16, 2016, 9:57pm

Hey Reuben,

I don’t understand the comment. what do you mean by “cloud write loop”?

From my perspective it seems that the rules file is actually causing the problem. I would think that the device should not send any info to the cloud unless I tell it to no?

rock

Reuben · August 16, 2016, 10:06pm

Some Linux distributions try to send AT commands or something when a serial port pops up because they think it’s a modem. If you do that with the Dash, then all those messages get sent to the cloud.

The rules file is supposed to stop that from happening so it’s odd that getting rid of it fixes the problem. I’m just looking to see if something else could be going on there.

The cloud write loop thing is just if you’re running your own user sketch which it sounds like isn’t case here so nothing to worry about.

rscavetta · August 16, 2016, 11:02pm

Ok, I understand, that explains the generated output, not sure why the rules are misbehaving though… strange… Maybe some idiosyncrasy with my distro or udev version… but not a showstopper anymore and it looks like I can continue testing for now.

Thanks for all your help guys, I’ll let you know if there are anymore issues.

rock