Dash SMS/OTA issues

Hi, Hologram,

I haven’t been able to push an OTA to the Dash for 2 days now. Is anything up with the OTA system? TonyM has the same issue.

OTA shows as having been sent from Arduino IDE (latest versions of everything, Boards Manager + Dash hardwire updated), but I never get the Bright White LED / sketch upload when trying OTA.

Thanks!

@MichaelM,

I’m having issues at the moment also. I’ll investigate and let you know what I find.

Thanks! (Always on the weekend, amirite?)

Hi there, looks like there was a misconfiguration on the server and a bug in our code that was breaking things. Those are both fixed so things should mostly be working, but @Erik thinks he found an issue in the firmware that could be causing intermittent problems that you may have been seeing before. He’s looking at it and we’ll put out a quick system firmware update once we get that fixed.

Sorry for the issues with this.

Thanks, Reuben! Will get to testing tonight / tomorrow and let you know. Enjoy the weekend!

Whatever you did, it worked! :slight_smile:

OTA now functional; saw Erik’s note as well, so sticking with 0.9.8 for now.

Thanks!

Hey try out the version we just posted. Should work better for OTA stuff

Thanks, Reuben,

Unfortunately, I’ve discovered what [may be] a bug in the new FW.

My sketch hangs on

HologramCloud.attachHandlerSMS(my_sms_handler);

Anything after this won’t run. I saw the same issue in 0.9.10. Thoughts?

We’re looking into that. It is probably related to some changes we made in the new firmware to queue up messages in the modem instead of using RAM on the system chip.

We haven’t been able to reproduce that issue on our side yet.

Does it always hang or just after programming? Does power cycling help?

Well, you are going to love this one:

I have 2 Dash units (early 1.1) up and running for development work. They get every update, and both have been frequently programmed OTA. All was well until last Friday, when the OTA system got flaky. I updated both boards over USB to 0.9.10. At that point, I recall that all was well (which is key for the next paragraph). I then tried another OTA, and that hung up both boards on sketches where the SMS handler was included. On a hunch, I downgraded both boards to 0.9.8, and VOILA – both working well again on the same sketch. Another upgrade attempt to 0.9.10, and the hang repeats. TonyM is seeing identical behavior (also Mac Arduino IDE) on his Dash 1.1.

Here’s where it gets interesting. I recalled that 0.9.10 initially seemed okay, and that Erik could not reproduce the issue, even with the SMS handler. On a hunch, I took a new virgin Dash from my “future development” inventory minutes ago. It was at FW 0.9.3 right out of the anti-static bag, (had the factory modem serial debug) and I updated that board directly to 0.9.11. Screenshot attached. The same sketch as I posted above, w/ SMS message handler, now works perfectly.

So there you have it. A new (virgin) unit, next to a well-used unit. Both are free standing (no breadboards). Both are 0.9.11, and both are running the same sketch (with SMS handler). The virgin board works great, the older board hangs on the SMS handler 100% of the time. Both are Dash 1.1 units. On the older board, hardware power reset (disconnect / reconnect) does not help; sketch always hangs. Reloading same sketch: no joy there, either. Removing the SMS handler from the sketch returns the older unit to normal function. Both units – jumper set to USB mode.

My system: Mac 10.12.5 Sierra, Arduino 1.8.2, Dash Board Manager 0.10.2. TonyM is also using a Mac, same SW levels, same hangup behavior on an early Dash 1.1 (long header pin version).

So, my inferred hunches, based on nothing other than above:

  1. The OTA on Friday or Saturday (pre-Reuben fix) did something to the FW (or boot loader?) that later FW uploads did not negate. Either that, or the sequential / incremental FW updates have a different effect at 0.9.11 than the 0.9.3 to 0.9.11 (leapfrog) update.

  2. There may be HW updates or changes to later 1.1 units since we received our early 1.1 units.

As I say, uninformed guesses, but the behavior of old, incrementally updated units = bad, and new, direct upgrade to 0.9.11 is solid. For what it’s worth…

New Dash unit on left happily blinks its LED. Same sketch on right unit – no Blinky. And yes, I did plug it in!

Any thoughts greatly welcome!


…Assuming no hardware changes, is there maybe a way to return the Dash to “Factory Fresh Condition” and try from there?

HI MichaelM,

I ran into the same issue. Digging into the code I realized that the attachHandlerSMS calls a modem command and indefinitely polls for events if there are any SMS in the QUEUE in the modem. Since I did not need backlogged SMS. I added the following code that fixed this:

#include "system/hal/ArduinoModem.h"
...
begin(){
  ...

  modem.command("+HSMSD=1,4"); //Delete all prior smses
}

This seems to work fine.

Hi, Srikanth,

I REALLY like the ability to insert AT commands directly into the code. Thank you for showing that!

I was able to implement your code into the sketch above, and into another one that is hanging on the same call. Unfortunately, it didn’t solve the problem for me. I can receive SMS’s without problem on the Dash, but my issue still appears to be FW related – 0.9.11 hangs, and 0.9.8 works fine.

Update… Well, looks like I have managed to bork my latest “factory fresh” Dash. A couple of OTA updates loaded the sketch just fine, but now the formerly good / new Dash hangs on the SMS handler under 0.9.11 just like the others. Damn! It has joined the Borg Collective. Locutus would be pleased.

Downshifting to 0.9.8 brings back the joy, but only after a power cycle. Note: I don’t even need to reload my sketch.

Update2:

Perhaps the issue is not directly related to SMS.

The following dead-simple program runs fine under 0.9.8.

It goes through the loop ONE TIME ONLY on 0.9.11, then stops forever. Neither reset button brings joy. Power cycle only, and same single loop iteration as before.

A puzzler, yes?

void setup() {  
Serial.begin(9600);
delay(5000); // Open serial monitor and watch the [non] action...
}

void loop(){
Serial.print("We are running the loop...\n"); 
delay(2000);
}

@MichaelM,

I haven’t been able to re-create your failure, but I’m trying. Would you add
Dash.pulseLED(500,500);
as the first line in your setup() and let me know if it continues to flash or not?

SMS polling happens on the HologramCloud.attachHandlerSMS(my_sms_handler); call and after each time through loop(), so that is still looking like the main culprit.

Thanks, Erik!

I put the Dash pulse statement as first setup command.

As I expected, it ran fine – nice pulsing LED – under 0.9.8, and though it pulsed continually under 0.9.11, the loop ran once (but the LED continues to pulse, so the board lives at some level…).

Here’s a screenshot of the sketch, and a screenshot of the serial monitor after 5 minutes under 0.9.11 (just so you see exactly what I’m seeing). 0.9.8 quickly fills the serial monitor with “We are running the loop…”

Update: Just to rule out any possible weirdness from my Mac IDE installation: I tried the same sketch from a different computer (also a Mac). Same results.

Update: Problem solved!

Many thanks to Erik at Hologram! He walked me through checking the Dash for rogue SMS messages, and then deleting them. The latest Dash FW (0.9.11) stores the messages in flash memory, so they stick around in the modem until actively deleted. Loading the Dash Passthrough sketch allows direct access to the modem. From there, it’s a matter of seeing if any messages are present (using the AT+CMGL command), then deleting them one-by-one (with the AT+CMGD command – don’t forget to learn about index numbers so you can delete all SMS’s).

In my case, there were multiple SMS messages hiding in the modem, and deleting them brought back full Dash functionality under 0.9.11. It looks like Hologram will continue to evolve their OTA method, so we can hopefully expect this issue to go the way of the Dodo Bird in short order.

Thanks again to Erik and Hologram for the great support and solution to my issue.

Hi Micheal,

That’s exactly the problem I was facing there were 7 messages on the modem flash and the attach handler was hanging because of that.

The main loop does not run more than once because of the code in pollEvents. That is run automatically at the end of each loop (according to the docs).

Can you share the code you are using to delete the Rouge smses? Just curious on your approach. The code I shared earlier managed to delete Rouge SMSes with just one command without knowing exactly the message index etc. I am particularly interested in learning about the passthrough.

When did you exactly call passthrough and how did you run the at commands was it different from my approach?
HologramCloud.enterPassthrough();
modem.command("+HSMSD=1,4"); //Delete all prior smses