Dash Modem Times Out and Won't Send Data After 1 Hour

MichaelM · March 6, 2016, 7:39pm

My Dash is working great when sending data to the Konekt Cloud at a frequency of once every ten minutes. When I reduce that frequency to 1x per hour, the modem seems to disconnect and not send anything, even though my Arduino sketch is working well (per the serial monitor) and attempting to send the data. I can provide sample code if needed; here’s a bit of the error debug I got when I extended the Hello_World sketch to send a message once per hour. The initial message went fine; subsequent messages errored out. The error is repeatable across numerous sketches – the error appears to be tied to the one-hour delay, although it may actually at intervals less than one hour.

Serial Output below…

`+EVENT:LOG,51,5,DEBUG,40,Bootstrapping (10/14): Link up (1 of 2).
+EVENT:LOG,51,5,DEBUG,40,Bootstrapping (11/14): Link up (2 of 2).
+EVENT:LOG,61,5,DEBUG,50,Bootstrapping (12/14): Link verification (1 of 2).
+EVENT:LOG,61,5,DEBUG,50,Bootstrapping (13/14): Link verification (2 of 2).
+EVENT:LOG,67,5,DEBUG,56,Bootstrapping (14/14): Enter serial passthrough mode. OK
+++
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
+EVENT:LOG,42,5,DEBUG,31,Modem event: Network registered
+EVENT:LOG,39,5,DEBUG,28,Modem event: Data registered
Sending a FIRST message to the Cloud…
FIRST Message sent!
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
Sending a message to the Cloud…
Message sent!
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,42,5,DEBUG,31,Data successfully sent to cloud
Sending a message to the Cloud…
Message sent!
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,26,5,DEBUG,15,Socket failure!
+EVENT:LOG,43,4,WARN,33,Socket failure during socket open
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,48,4,WARN,38,Failure sending data during cloud push
+EVENT:LOG,38,4,WARN,28,Cloud send failed! Retrying.
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,44,5,DEBUG,33,AT+USOCO=0,“23.253.146.203”,9999
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,26,5,DEBUG,15,Socket failure!
+EVENT:LOG,43,4,WARN,33,Socket failure during socket open
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCL=0

MichaelM · March 8, 2016, 3:32pm

UPDATE: Assuming the modem is having sleepy time after an hour, I modified my sketch to send a “keep awake” signal, (consisting of a single space) every thirty minutes. The space (4 bytes!) is too short to trigger the SMS forwarding rule (yay!), but of course I’m burning data with that, not to mention that it seems the battery is losing ground when powered from my PC (boo!). My 2A USB adapter fixed the net energy loss situation, but of course I lost the serial monitor.

So, it looks like there is a modem timeout, possibly to save power (and also maybe data overhead?). Any word on when we might be getting those super-slick sketch / modem sleep / wake commands? Thanks!

MichaelM · March 17, 2016, 4:31pm

UPDATE2: The modem now seems to be timing out quicker than before; but more testing is needed to verify. My 30-minute keep-awake ping is no longer keeping cellular data things awake for data upload. Only a complete power cycle of the Dash appears to re-boot the data connection. Reloading the sketch (power on) seems to leave the modem offline.

UPDATE3: My original notes on the “net energy loss” in the UPDATE post above appear incorrect. After more testing, it appears the Dash is charging fine via the always-weak computer USB (or via my 2A USB charger). The energy issue actually appears to be a bug in the Dash charger: My USB power meter shows the Dash consistently drawing 0.5A (or slightly less). The battery will always charge to 100%, but as soon as it hits that, it slowly loses net charge until it reaches 0%, at which point the Dash shuts down (yes, even when plugged into 2A USB power). I’m using the Dash class to report State of Charge to the serial monitor, so it’s easy to track.

Reuben · March 18, 2016, 3:53am

Are you saying that if you charge it with less current then it works correctly?

MichaelM · March 18, 2016, 4:15am

No, actually – and very sorry for the confusion… I originally thought, since I was seeing decreasing battery levels after running a long time in battery + USB mode, that my charging voltage was too low. Changing to a 2A charger brought the battery back to full charge; this took a while to figure out, since putting the Dash on a dedicated 2A charger meant that I lost the serial monitor. So, to read battery level, I uploaded the battery SOC to Thingspeak @ 1-hour intervals. That’s when I discovered what was really happening: On both of my Dashes, on either the PC USB or dedicated 2A charger, the battery will fully charge to 100 - 101%, but then it tapers off 4-5% per hour after that until the battery reaches 0% and the Dash shuts down. Here’s a screen shot of the SOC at 1-hour intervals. Note: The first low point, just above the 08:00, is a power-cycle reboot to see if the battery would again start to recharge. As you can see, it did. It than ran for a day or so until it reached zero and shut the Dash down. I am never seeing more than 0.45 - 0.47A / 5.05V at the USB.

Reuben · March 18, 2016, 4:18am

Ah ok, yeah, that’s bad. Cool graph though. We’ll keep looking into that.

MichaelM · March 18, 2016, 4:30am

Ha! I thought I was going crazy-bananas – that’s why I made the chart, to prove to myself as much as anything. The rest of the sketch just sends a few fields of data to Thingspeak once per hour, plus my once-every-thirty-minutes keep-awake ping effort for the modem. And, no, I’m not trying to power a space heater from the analog pins or anything nuts like that… It seems, from an outsider’s perspective, that the battery charges fully, then the charge maintenance routine fails. Power cycling the Dash puts it back into charge mode, but as soon as it’s fully charged, down we go again. Like watching a goldfish come up for food to the surface, then slowing moving back into the depths of his little bowl. I’ve repeated this more than a few times now. Sadly, this means the Dash can’t be operated with the battery + USB mode for more than 48-72 hours or so. (I’m using the 2,000 mAh battery from Konekt – it’s a decent little cell!) Always grateful for your efforts!

FrankVigilante · March 23, 2016, 2:18am

I am having a similar issue, but all of the sudden my Dash won’t connect at all. It has worked great for the past two weeks, but today it just seems to be acting funny. Serial reports below:

+EVENT:LOG,15,5,DEBUG,5,30000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,20000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,10000
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:

The timeout seems to jam up every attempt at bootstrapping. Some unsuccessful steps I have taken to fix it:

Powered over Battery to ensure enough current getting to the Ublox
System Reset / User Reset buttons…etc
Switched back to 0.7.1 in the Arduino IDE
Upgraded antenna
Tried multiple sketches that have worked in the past

Any ideas would be much appreciated…My Dash wants to get back in the cloud.

Reuben · March 23, 2016, 2:25pm

Hi Frank, I think you might be having a different issue. That actually looks like your board died and can’t initialize the modem anymore. Can you email us at support@konekt.io and we’ll see what we can do?

jasoveen · April 4, 2016, 6:20pm

I am also seeing the same issues as MichaelM. When I try to send a message to the cloud after an hour I get the same failure messages.
+EVENT:LOG,38,5,DEBUG,27,Opening connection to cloud
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+USOCR=6
+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,60,4,INFO,50,An error occurred while processing a modem command

I’m also seeing the battery slowly discharge after reaching 100% charge (it’s also plugged into a 2.5A USB hub connected to my computer).

AlexS · April 15, 2016, 6:17am

I’m also having the same problem. My modem was left on for over an hour and after receiving a notice about network registration (don’t recall the exact wording, didn’t get a screen cap) my modem no longer works despite power cycling or reseating the SIM/antenna. All I get is the output below. Has anyone had luck with a fix?? I have a time-sensitive project coming up that needs the modem functioning.

+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,20000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,10000
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4
+EVENT:LOG,31,4,WARN,21,Modem command timeout

HologramPat · April 15, 2016, 8:13am

Hey everyone,

FYI, we’re going to have a release candidate firmware image soon containing self-healing improvements that might help with this timeout issue.

If I posted a link to a pre-release version, could some of you here repeat experiments with it and let us know how the behavior changes?

Thanks,
PFW

AlexS · April 15, 2016, 11:45am

Hey Pat, I’m definitely willing to give it a shot!

MichaelM · April 15, 2016, 2:12pm

Absolutely yes, Pat. I’ll have it up and testing soon after you post the files, and of course I’ll have the feedback to you as it happens. Much appreciated.

AlexS · April 20, 2016, 2:17pm

Hey Pat, do you have any update on when we might be able to get our hands on the pre-release firmware? I have a PoC demo coming up and need to know if I’ll have to order a Fona or something for short-term testing until this gets sorted out.

Thanks again for the help though! I’ll be glad to provide any feedback I can on the results.

HologramPat · April 20, 2016, 4:52pm

Hey @AlexS,

Should be this week. We have a test bed set up where we were able to reproduce the issue, which is a huge help in diagnosing. Will keep you posted!

Thanks,
PFW

jasoveen · May 4, 2016, 10:34pm

Any updates on this new firmware, it’s been over two weeks?
Also, what about the battery slowly discharges after reaching 100% charge issue?
I’m also seeing the issue where the first message sent never shows up on the dashboard (already discussed in another thread)

These seems like such basic use cases it’s crazy to still be having these problems this far along with this product.
I’m really starting to loose faith in the entire Dash product.

AlexS · May 5, 2016, 6:18am

Hey Everyone,

I actually got my modem to recover from the “Modem Command Timeout” loop that happens when you let the modem sit idle for over an hour. My Dash Pro was stuck in this state even after power cycling and letting sit for a week. I’ll start by saying that this could’ve been just dumb luck but it’s worth sharing. Here’s what steps I played with and eventually got it to work:

I re-loaded the firmware (0.9.2) onto my Dash Pro, still got the same error.
Powered it via a powered USB hub with an in-line power meter, same timeout errors except the power meter would reset once every few seconds. (This led me to believe it was attempting to draw massive spikes of current, too much for even my powered hub).
Next I put it on a breadboard, disconnected all power, and shorted the power & gnd rails (my though is that this will remove any latent charge from on-board caps and ensure a power reset).
Powered the board from a bench-top power supply with 5V capable of many amps, let that sit for a minute by chance, no terminal so I couldn’t see if anything was happening.
I then disconnected the PSU and connected it to my computer (directly with no hub or anything in-line) and boom, it booted right up and allowed me to send & receive messages as usual.
Finally I moved it back to the USB port on my monitor’s built-in hub via an in-monitor hub (as I used to do successfully before the timeout failure) and it continued to chug along sending and receiving messages.

At this point i’m going to just make sure that it never sits idle for too long (either reset it or send a status message) simply because I need this for a demo soon and don’t have the time to deal with a proper “fix”. But if it locks up again maybe i’ll take a more experimental approach to the fix. Please let the thread know if any of these steps do or don’t help you out.

Just a final note to everyone, I do honestly agree that this product feels lacking in support and proper release-day features but i’m certain they’re squashing bugs as fast as they can. The more supportive we are in terms of debug information and willingness to experiment the sooner this can get to the best end-product state possible.

bitshift · May 5, 2016, 2:00pm

I’ve seen the first dropped message problem too. We’ve just accepted it as the sacrificial lamb to get the connection going (I also have our device watching for ‘passthrough’ as a keyword from the cloud uart to determine when we can start sending data).

We are also waiting for the modem fix too. Actually, is there a low level GPIO pin we can access to trigger the modem/system IC to reset?

MichaelM · May 9, 2016, 8:01pm

Hi, Pat,

Just your friendly Monday PING! to see how the firmware update is coming – pre-release or otherwise! Eager to start testing long-term modem performance.

Thanks!