Dash Modem Times Out and Won't Send Data After 1 Hour

Hey Michael,
Pat’s been in constant meetings this week so hasn’t been able to reply but we’re trying to get some cool updates out the door by Friday and hopefully this will be included. More information soon.

3 Likes

Hey again, I know we missed that deadline. We’re trying to iron out a couple more issues in that firmware update so it’ll be a little longer. We have some other updates to the general platform and arduino integration coming very soon though. (Including stuff like analogWrite support and fixes to USB driver issues.)

Hi Ruben,
Can you give some sort of ETA for the new firmware. I’d prefer a realistic estimate than an optimistic one. I realize these types of releases can take a huge amount of time and effort - so this isn’t a complaint, I’d just like to make some plans on the basis of having the new firmware available.

Hey @ajandar @MichaelM @bitshift @AlexS @jasoveen @FrankVigilante,

I can give an update.

The problem:

  • On some cellular towers for some carriers, the PSD context (i.e. PPP) was being dropped after inactivity
  • This caused strange effects to the modem’s internal state machine, which were not being reset as expected when trying to immediately self-heal
  • Seemingly not every carrier/tower exhibited this behavior in the same way (strange, no?); nonetheless, it was a widely reported issue

The potential solutions:

  1. Send a keep-alive packet every so often (not acceptable, since that would waste cellular data)
  2. Get the modem module to renegotiate properly if/when this occurs (desirable solution)

Option #2 proved difficult to implement reliably. We added a number of new states into our state machine and tests to verify if/when the modem module was not in a state where the PSD context could be recovered without requiring a deeper reset.

We have successfully reproduced the issue, and have just finished a 0.9.4 release candidate (after needing to perform extensive testing of the additional states in our state machine). Official release will be early next week, but we’ll post the release candidate here prior to that.

In firmware version 0.9.4, we implement Option #2: in the event that the carrier drops the PSD context, and the modem module enters a state where a more significant renegotiation must occur, we detect that and correct the state of the modem module. This improves the robustness of our self-healing code. No keep-alives are necessary.

Also in version 0.9.4: We identified a socket-level networking stack issue that, under certain circumstances, would cause an error connecting to a server the first time a connection is attempted after initial module boot. We have been able to reproduce that issue. We have implemented a workaround for that issue within our modem driver, which is also included in version 0.9.4.

Best,
PFW

Hey @ajandar, @MichaelM, @bitshift @AlexS, @jasoveen, and @FrankVigilante,

Here is a release candidate (use at your own risk) System Firmware Version 0.9.4 that addresses these issues: http://downloads.konekt.io/dash/system_firmware/dash_system_module_firmware_0.9.4.bin

Main changes:

  • Implemented a mechanism to repair modem module internal state if it gets hung under certain circumstances immediately following a forced PSD context timeout by a carrier/tower
  • Implemented a workaround for a separate bug internal to the third-party modem module, where, under certain circumstances, the modem might not connect to a server on the first attempt

Please let us know your experiences, and thank you for your patience.

Best,
PFW

Thanks - is this just Dash firmware or also OK for Dash Pro?

@ajandar yep, 0.9.4 RC is for the Dash Pro too.

I’ve loaded the new firmware - but still get the following;
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&F0
&
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,240000

That looks fine. Did you wait the 30 seconds for the modem to re-initialize and connect?

Look for the keyword “Serial passthrough” to determine if the modem is ready to accept data. I added logic that waits until I see that keyword before I let the application send data (among others until we get an API for it).

1 Like

I tried the 0.9.4 firmware on my Dash Pro and I am also getting the same errors as @ajandar when it starts up and tries to connect to the cell network.

If I install 0.9.3 it’s able to connect to the cell network, but still seems to loose the first message sent.

Below is what I get:
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&F0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,22,5,DEBUG,11,AT+CFUN=16

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,60000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,50000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,40000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,30000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,20000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,10000
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&F0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&F0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&K4

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,15,5,DEBUG,5,ATE0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,15,5,DEBUG,5,ATE0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,15,5,DEBUG,5,ATE0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,15,5,DEBUG,5,ATE0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,120000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,110000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,100000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,90000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,80000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,70000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,60000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,50000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,40000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,30000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,20000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,15,5,DEBUG,5,10000
+EVENT:LOG,81,5,DEBUG,70,Bootstrapping (1/14): Init modem (can sometimes take several minutes).
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&F0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,98,4,CRIT,88,Insufficient b+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&F0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,25,5,DEBUG,14,Modem command:
+EVENT:LOG,16,5,DEBUG,6,AT&F0

+EVENT:LOG,31,4,WARN,21,Modem command timeout
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,180000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,170000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,160000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,150000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,140000
+EVENT:LOG,50,5,DEBUG,39,Retry delay before modem bootstrapping:
+EVENT:LOG,16,5,DEBUG,6,130000

Does this happen when you pull the power and then restart it?

Yes, I removed the power and the battery a couple of times. I received the same errors each startup.

Hello, All,

Well, I’ve been testing the Dash 0.9.4 Release Candidate, and I’m sorry to report it’s a bit of a mess in need of further improvement. Here’s the details:

My setup: Mac El Capitan, 10.11.5. Dash (non-pro model) @ 48Mhz. Arduino IDE 1.6.7, Board Manager updated to Konekt Dash/Dash Pro 0.9.0. No battery / USB only.

Results:

  1. “Hello, Cloud” sketch. Bad behavior in both warm boot and cold boot modes, per below:
    a. The Dash will do its usual “slow” connect to the cell network on a warm boot (sketch upload). It then only uploads a single “Hello, Cloud!” before the modem fails. It seems to try valiantly to reset, and does manage (sometimes!) to send a second or even a third “Hello, Cloud!” after a delay of 2-3 minutes. Log shows 6 sent, Dashboard shows only 3 received. Sadness.
    b. The Dash WILL connect to the cell net more quickly on cold boot as with earlier FW versions (power off, then re-connect). Step 14/14 reached much more quickly than with warm boot upload. The first message arrives corrupted as “\0Hello, Cloud!” Sometimes I will get 2 messages, sometimes as many as 5. I have never received all 6. Note: The backslash-zero addition is only on cold boot.
    c. I’m using Pat’s 0.9.4 Release Candidate from this thread, but the FW version is shown as 0.9.3 with M2 button press / via serial monitor. Drove me nuts until confirmed by TextEdit view of 0.9.4.bin file that it’s still called 0.9.3 internally. Sigh.

So, there you have it, at least with this sketch. Some of the logs say fairly ominous things like “modem did not power on.” Later in the log, I can see that it looks like it may have indeed turned on, but by then the sketch had timed out on the number of sends.

I’ll try one of my sketches that’s meant to keep running for days, and see how that goes. I’ll report back as the story develops.

Thanks for the feedback. We’ve identified that there are some intermittent issues occurring with 0.9.4-RC1 and are trying to scrub through them as fast as possible. One concern is the power being supplied to the Ublox module. Any information about how your Dash/DashPro is powered would be greatly appreciated, including cable type (regular micro usb, or Y-cable), approximate cable length, whether you’re connected to a powered USB hub, etc.

Also whether you’ve been restarting the Dash/DashPro by unplugging/replugging, hitting M2 reset, simply letting it reset after firmware update, etc.

I have my Dash Pro powered via a USB 2.0 powered hub (with a 2.5A plug) using the supplied 1ft USB cable and a 3.7v 2000mAh battery (with the jumpers set for battery). No matter how I reset the Dash by unplugging/replugging, hitting M2 reset, or by uploading new firmware/user program I get the “Modem command timeout” errors when using 0.9.4 and it never connects. If I go back to the 0.9.3 firmware it does connect to the cell and sends messages with the original issues.

My Dash Pro also has the bug/issue where once the battery seems fully charged it starts draining the battery even with the USB still connected. I have to unplug the battery and USB to reset it. It will then recharge the battery and repeat the discharge. It will never charge the battery and maintain the charge.

I also tried 0.9.4 with the battery disconnected and the jumpers reset to the non battery mode and I still get the “Modem command timeout” errors and no cell connection.

Is my Dash Pro defective? In the current state this thing is worthless since it can’t even perform the basic task of sending data reliably and charge/maintain a battery.

More feedback from overnight testing:

My Dash, running a “always on” sketch that is meant to upload a data set to Thingspeak every hour, plus a “keep-awake” ping every 30 minutes did not run well. The data uploaded at the prescribed intervals for 3 hours, then ceased overnight (9 hours without data). At 7 AM this morning, the data stream resumed, so there IS some type of self-healing underway; just not rapidly enough to be practicably differentiable from a total system failure, unfortunately.

Dash is powered directly by newish Gigabyte mobo (Z97) which is also my serial monitor port. Nothing else connected on that USB channel; no power issues noted. My sketch incorporates an elapsed time counter (readable in my serial output) that shows the Dash has not reset, so the data outages here happened while the Dash was running uninterrupted. M2 button unmolested as well. Battery NOT connected; happy to retry with battery as this gives nice, steady DC power with decent peak amp reserve; suspect the data stream will fail before the charging bug drains the battery.

UPDATE: Powered with Battery + USB. Cellular failure after about 2.5 hours. Battery at very near 100%, jumpers set for batt+USB power. Continuing to run sketch; I will advise if the modem comes back on line.

Thanks for the details, this is helpful.

There is a battery charging issue that we’ve identified to happen on certain boards. We apologize for this inconvenience, and are in the midst of updating our hardware design to fix this. We will be replacing these boards free of charge. I’ve noted you down as needing a replacement - we will come in contact for shipping details soon.

@MichaelM you are the man! I keep lurking to see if this is getting solved. Let me know if you want me to corroborate any tests. I don’t know if my board has a battery problem or not BTW.

1 Like