SMS Incoming Messages Delayed or Not Delivered to Dash

Ryan,

The pipe seems to be flowing again. It just starting working on its own.

Thanks for listening…

  • rock

Hey Ryan,

Well, we’re still having trouble with multiple dash devices just “going out to lunch”. Meaning they seem to receive SMS OK but anything sent to the cloud never gets there…

Below is the output from the dash when we see this behavior. After the Debug msgs stop and we send a SMS to the device it is received OK, but then still cannot send to the cloud unless we disconnect the Dash and reconnect.
It is happening when the device is inactive overnight. It is currently connected to desktop development box which is Ubuntu 14.04.4 LTS. The entire box is solid state and never goes to sleep (that I can see).

Note that I have tried two different Dash devices with the same symptoms which would seem to indicate my environment, but I’m not sure what to look at first. i would appreciate any ideas on how to debug this.

$ cat < /dev/ttyACM0
NT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,26,5,DEBUG,15,Socket failure!
+EVENT:LOG,43,4,WARN,33,Socket failure during socket open
+EVENT:LOG,60,4,INFO,50,An error occurred e m+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,48,4,WARN,38,Failure sending data during cloud push
+EVENT:LOG,38,4,WARN,28,Cloud send failed! Retrying.
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,26,5,DEBUG,15,Socket failure!
+EVENT:LOG,43,4,WARN,33,Socket failure during socket open
+EVENT:LOG,60,4,INFO,50,An error occurredwic+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,48,4,WARN,38,Failure sending data during cloud push
+EVENT:LOG,38,4,WARN,28,Cloud send failed! Retrying.
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,26,5,DEBUG,15,Socket failure!
+EVENT:LOG,43,4,WARN,33,Socket failure during socket open
+EVENT:LOG,60,4,INFO,50,An error occurred eaa+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,48,4,WARN,38,Failure sending data during cloud push
+EVENT:LOG,38,4,WARN,28,Cloud send failed! Retrying.
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,26,5,DEBUG,15,Socket failure!
+EVENT:LOG,43,4,WARN,33,Socket failure during socket open
+EVENT:LOG,60,4,INFO,50,An error occurred sm+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,48,4,WARN,38,Failure sending data during cloud push
+EVENT:LOG,38,4,WARN,28,Cloud send failed! Retrying.

  • rock

Hi Rock,

Can you try updating to the latest RC firmware 0.9.4?
https://hologram.io/docs/hardware/hologram-dash/downloads/

This looks like a modem timeout issue we addressed in the latest release. Let us know if you have any questions or still encounter issues sending outbound SMS from the device when inactive overnight.

Thanks,
Ryan

Hey ryan,

good timing… I am just trying that now as I found the other threads, but the dashupdater v0.6.2 is failing.

I have tried a few times now with no success.

My procedure:

  1. With the device working, i press the PGM button and the device does enter program mode; i have the correct LED flashing pattern

  2. run the dash updater

  3. select the new fw file (konekt_system_module_dash_system_0.9.2.bin)

  4. select USB connection

then I get the failure message.

note that the device is attached and working

This is a Ubuntu 14.04.4 LTS desktop system.

anything else I can try?

  • rock

Can you try with the latest dash updater utility v0.7.1?
https://hologram.io/docs/guides-tutorials/using-our-platform/tutorials/upgrade-dash-user-program-and-firmware/#hologram-dash-updater-utility

This may also be worth looking at: DashPro not connecting to the Ubuntu USB controller - #10 by Reuben

Hey Ryan,

yea… I tried that version already. same error.

i read some threads re: power issues so I put the Dash on a powered hub and the results are the same. Still has the issue.

I’m going to try updating the FW via windows to see if that works but I’ve seen other threads that indicate issues with that version as well. i’ll post the results.

Reuben:
I saw that thread and installed the udev rule but all that did was send the boot messages to the cloud. it still won’t send SMS to cloud after a period of inactivity.

  • rock

Hey, just wanted to make sure you saw my reply right above. There could be a permissions issue in Ubuntu that can be fixed by downloading a rules file.

OK, can you provide details? What are the permissions issues?

  • rock

udev might be creating the Dash /dev node with root permissions only. If you get the rules file at https://raw.githubusercontent.com/hologram-io/hologram-dash-arduino-integration/master/85-hologram.rules and copy it into /etc/udev/rules.d then it might fix your issue.

Hey Reuben,

Ok… kewl…

The current permissions are:

crw-rw---- 1 root dialout 166, 0 Aug 10 15:50 /dev/ttyACM0

I guess they should be:

crw-rw-rw- 1 root dialout 166, 0 Aug 10 15:50 /dev/ttyACM0

and the udev rule will make it as such. Is that correct?

which part of that rule sends all boot messages to the cloud? I would like to turn that off so I’m not paying for it

  • rock

Oh, so installing that rule also prevents Linux from trying to send a bunch of AT commands that would just end up in the cloud.

It’s actually one of the /dev/hid* or /dev/usb/hid* nodes that get used for programming. ttyACM0 is just used for the serial communication.

If you’re seeing debug messages from the Dash itself going to the cloud then that would be something in your sketch.

Hey Reuben,

the permissions thingy fixed the firmware upgrade problem; I successfully upgraded to 0.9.2

I will monitor the hang issue overnight and post back. thanks for the help.

  • rock

Hey Guys,

I have been able to upgrade the FW to 0.9.2 but still have the same issues on two different dash devices. We are using only the device str8 out of the box with no custom sketch or code involved. I have been moving a SIM card between them for testing as I only have a single SIM activated for now; I don’t believe this could have this effect (is this a valid assumption?)). The behavior is as follows:

Bidirectional SMS worked for a short time over a few hours. Then i left the device idle for ~30 minutes and when I tried to send the device a message via REST API I get the below output from the device. It has been truncated below to save space but the messages repeated for several minutes before the “Cloud send failed! Retrying” message at which point the message came through. But… while it recovered on the receive side, I still cannot send SMS from the device to the cloud.

+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,26,5,DEBUG,15,Socket failure!
+EVENT:LOG,43,4,WARN,33,Socket failure during socket open
+EVENT:LOG,25,5,DEBUG,14,Modem command:ed rm
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,48,4,WARN,38,Failure sending data during cloud push
+EVENT:LOG,38,4,WARN,28,Cloud send failed! Retrying.

// msg finllay arrives from API
+EVENT:SMSRCVD,112,2016-08-11T16:19:14.5223963Z|command:SET_ATTENUATOR,attenuatorLevel:int,time:ISO 8601 date/time,signature:byte[]

Not sure what to do now… Are you guys still working on the FW for this issue or is 9.2 supposed to have fixed it?

Also, how can I verify what firmware is actually loaded on the device? The updater said it loaded but I see no way to verify

  • rock

Rock - Sorry saw you’re using 0.9.2. The fix for this is in the current release candidate 0.9.4 of the firmware. Can you verify with that as well? Sorry you’ll have to load another set of firmware.

The firmware version can be checked by pressing the tiny M2 reset button while snooping the serial port (via USB or TTL). This will reset the system module and print the firmware version for you. On the Dash, the M2 reset is just a little bit below the antenna connector.

Hey Ryan,

I found 0.9.4 here: http://downloads.konekt.io/dash/system_firmware/dash_system_module_firmware_0.9.4.bin from this thread: Dash Modem Times Out and Won't Send Data After 1 Hour - #24 by HologramPat.

Is that the correct binary? If so, it does not work either. However the problem is now the reverse. We can send SMS to the cloud but we cannot send from the cloud (or API) to the device. Below is the output from the device. I used the M2 switch and verified the version only the device hung and would not complete booting (I waited 15 minutes). After I disconnected/reconnected and the device came up I could send to the cloud but not the reverse.

Output after using M2 switch:
$ cat < /dev/ttyACM0
+EVENT:LOG,37,5,DEBUG,26,Konekt Dash system init…
+EVENT:LOG,15,5,DEBUG,5,0.9.4
+EVENT:LOG,47,5,DEBUG,36,Beginning modem bootstrap process…
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&F0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&F0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+CFUN=16
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,76,4,INFO,66,Data connection deactivated by network, possibly due to inactivity�+EVENT:LOG,37,5,DEBUG,26,Konekt Dash system init…
+EVENT:LOG,15,5,DEBUG,5,0.9.4
+EVENT:LOG,47,5,DEBUG,36,Beginning modem bootstrap process…
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,36,5,DEBUG,25,Modem self-tests complete
+EVENT:LOG,46,5,DEBUG,35,Bootstrapping (2/14): Config modem.
+EVENT:LOG,47,5,DEBUG,36,Bootstrapping (3/14): Init SIM card.
+EVENT:LOG,33,5,DEBUG,22,Modem event: SIM ready
+EVENT:LOG,23,5,DEBUG,12,IMSI loaded.
+EVENT:LOG,24,5,DEBUG,13,ICCID loaded:
+EVENT:LOG,30,5,DEBUG,19,8944501006151414358
+EVENT:LOG,23,5,DEBUG,12,IMEI loaded:
+EVENT:LOG,26,5,DEBUG,15,353162070596656
+EVENT:LOG,53,5,DEBUG,42,Bootstrapping (4/14): Init event handling.
+EVENT:LOG,57,5,DEBUG,46,Bootstrapping (5/14): Wait for carrier signal.
+EVENT:LOG,36,5,DEBUG,25,Modem event: signal found
+EVENT:LOG,57,5,DEBUG,46,Bootstrapping (6/14): Establish operator link.
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,38,5,DEBUG,27,+COPS: 0,0,“AT&T”,2

OK

+EVENT:LOG,48,5,DEBUG,37,Bootstrapping (7/14): Config network.
+EVENT:LOG,40,5,DEBUG,29,Modem event: Other (ignored):
+EVENT:LOG,32,5,DEBUG,21,+UPSND: 0,8,0

OK

+EVENT:LOG,44,5,DEBUG,33,Bootstrapping (8/14): Config APN.
+EVENT:LOG,51,5,DEBUG,40,Bootstrapping (9/14): Config IP address.
+EVENT:LOG,51,5,DEBUG,40,Bootstrapping (10/14): Link up (1 of 2).
+EVENT:LOG,51,5,DEBUG,40,Bootstrapping (11/14): Link up (2 of 2).
+EVENT:LOG,102,4,CRIT,92,Modem instruction aborted mid-instruction. This most likely is a bug and should be reported.
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,41,5,DEBUG,30,Modem event: PSD context event
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,24,5,DEBUG,13,AT+UPSDA=0,3
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,65,5,DEBUG,54,Delaying to prevent carrier frustration, please wait:
+EVENT:LOG,13,5,DEBUG,3,30s
+EVENT:LOG,65,5,DEBUG,54,Delaying to prevent carrier frustration, please wait:
+EVENT:LOG,13,5,DEBUG,3,20s
+EVENT:LOG,65,5,DEBUG,54,Delaying to prevent carrier frustration, please wait:
+EVENT:LOG,13,5,DEBUG,3,10s
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,98,4,CRIT,88,Insufficient buffer while processing modem events, oldest event(s) dropped to make space
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,11,5,DEBUG,1,
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,46,5,DEBUG,35,Bootstrapping (2/14): Config modem.
+EVENT:LOG,47,5,DEBUG,36,Bootstrapping (3/14): Init SIM card.
+EVENT:LOG,23,5,DEBUG,12,IMSI loaded.
+EVENT:LOG,24,5,DEBUG,13,ICCID loaded:
+EVENT:LOG,30,5,DEBUG,19,8944501006151414358
+EVENT:LOG,23,5,DEBUG,12,IMEI loaded:
+EVENT:LOG,26,5,DEBUG,15,353162070596656
+EVENT:LOG,53,5,DEBUG,42,Bootstrapping (4/14): Init event handling.
+EVENT:LOG,57,5,DEBUG,46,Bootstrapping (5/14): Wait for carrier signal.
+EVENT:LOG,57,5,DEBUG,46,Bootstrapping (6/14): Establish operator link.
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,22,5,DEBUG,11,+COPS: 0,0,
+EVENT:LOG,40,5,DEBUG,29,Caller-listening modem event:
+EVENT:LOG,38,5,DEBUG,27,+COPS: 0,0,“AT&T”,2

OK

+EVENT:LOG,48,5,DEBUG,37,Bootstrapping (7/14): Config network.
+EVENT:LOG,40,5,DEBUG,29,Modem event: Other (ignored):
+EVENT:LOG,32,5,DEBUG,21,+UPSND: 0,8,1

OK

+EVENT:LOG,44,5,DEBUG,33,Bootstrapping (8/14): Config APN.
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,39,5,DEBUG,28,AT+UPSD=0,1,“apn.konekt.io
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,39,5,DEBUG,28,AT+UPSD=0,1,“apn.konekt.io
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,39,5,DEBUG,28,AT+UPSD=0,1,“apn.konekt.io
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,39,5,DEBUG,28,AT+UPSD=0,1,“apn.konekt.io
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,39,5,DEBUG,28,AT+UPSD=0,1,“apn.konekt.io
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,65,5,DEBUG,54,Delaying to prevent carrier frustration, please wait:
+EVENT:LOG,13,5,DEBUG,3,60s
+EVENT:LOG,65,5,DEBUG,54,Delaying to prevent carrier frustration, please wait:
+EVENT:LOG,13,5,DEBUG,3,50s
+EVENT:LOG,65,5,DEBUG,54,Delaying to prevent carrier frustration, please wait:
+EVENT:LOG,13,5,DEBUG,3,40s
+EVENT:LOG,65,5,DEBUG,54,Delaying to prevent carrier frustration, please wait:
+EVENT:LOG,13,5,DEBUG,3,30s
+EVENT:LOG,65,5,DEBUG,54,Delaying to prevent carrier frustration, please wait:
+EVENT:LOG,13,5,DEBUG,3,20s
+EVENT:LOG,65,5,DEBUG,54,Delaying to prevent carrier frustration, please wait:
+EVENT:LOG,13,5,DEBUG,3,10s
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,98,4,CRIT,88,Insufficient buffer while processing modem events, oldest event(s) dropped to make space
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command

The devices are not currently usable.

What I need to know: are your developers activity working on this and what is the ETA for a fix. Are there older versions of firmware available and if so where are they? Maybe we can find one that works.

  • rock

Hi @rscavetta,

Can you confirm how you are currently powering the Dash, and your jumper setting?

If you are encountering the issue you have described here, and if power cycling the device does not resolve the issue, this is most commonly caused by a power-related issue. The errors you’re seeing are indicating that the on-board baseband modem is not responding; since it is the component that consumes the most power out of all components on the Dash, insufficient power is the most common cause for the baseband modem to not respond to commands.

Version 0.9.4 of the system firmware includes additional in-field connection healing capabilities, baseband modem recovery functionality, and various bugfixes and feature improvements. One of the improvements concerning connection healing will reset or (if necessary) power-cycle the baseband modem after “Delaying to prevent carrier frustration” if the baseband modem does not respond to commands, in case of a low-power condition that has since been resolved (e.g. more power is suddenly available, battery has been recharged, etc.).

Your log printout indicates that, even after attempting to power cycle an unresponsive baseband modem, it remained unresponsive; additionally, while initially still responsive, the baseband modem began encountering errors coinciding with transmission and with bringing up the packet-switched data context (data service), just before becoming unresponsive. Usually we see this in cases of a power supply issue, as that moment in the boot-up process contains a lot of transmit activity.

Complete list of system firmware downloads is available here: https://hologram.io/docs/hardware/hologram-dash/downloads/firmware/

Best,
PFW

Hey Pat,

We have three physical devices. We use them in serial pass through mode and the power jumpers are correct and are just as shipped to us. They are powered via USB only. They are all acting the same way with FW 0.9.4.

I added a powered hub and it provides a constant 5.1 volts on the device and it seems to draw no more than 150ma when the activity light is flashing and idles at between 10 and 50ma.

The symptom does not change, it’s the same as the log I last posted. When it does works it works only in the device->cloud direction. i still cannot get a SMS to the device from the dashboard or the API. the old FW worked fine and only occasionally would hang when left idle for a while. Now it just doesn’t work at all in the API->device direction.

I’m willing to try any other suggestions you may have.

  • rock

Rock - Our carrier network partner experienced an outage yesterday (Aug-11) that affected inbound device terminated SMS messages. This was resolved as of this morning, have you been able to successfully send an SMS from API->device today?
As for the timeout issue, we still have an engineer investigating this but suspect it may be related to hardware issues on some earlier Dash hardware units. If interested, we could ship you a replacement unit when we get our 1.1 versions in the next week or two to check.