Recent Hologram change apparently broke SMSs to SIMs using API

Connectivity SIM Cards
1

Hoping @Reuben can chime in on this. As you know, I’ve been using Hologram for quite a while. I have a few dozen customers now and on Friday one of them reported seeing something that suggested that my product was receiving SMSs, but that it did not understand them. I saw the same. I finally got to investigate this today and the SMSs via the API are showing up at my cellular module differently than they ever have before.

When I send SMSs from my personal cell phone to the non-geographic +882… phone number, SMSs show up to the cell module normally. When I use the API (or using the “Messaging” page on the Dashboard) it is broken. I’ve changed nothing on my end. I have to assume this happened as part of the work on Friday, April 1.

xxxxx = redacted for privacy

message 01: “a”
message 02: “abc”

note:

1.timestamp in AT+CMGR response is an hour different when sending through +882… vs API
2.“from” number is always my personal 512xxx cell number. “to” number is different which is not really surprising, but may be of interest

----sending SMS from my cell phone to +882 36 00099 xxxxx

22:37:25.586 → AT+CMGR=0
22:37:25.700 → +CMGR: “REC READ”,“+1512xxxxxxx”,“”,“22/04/04,03:37:23+00”,145,4,0,3,“+19703769316”,145,1
22:37:25.774 → IN : a
22:37:25.808 → —> AT+CMGF=1
22:37:25.846 → <— OK
22:37:25.884 → —> AT+CMGD=000
22:37:25.918 → <— OK

22:37:35.142 → AT+CMGR=0
22:37:35.284 → +CMGR: “REC READ”,“+1512xxxxxxx”,“”,“22/04/04,03:37:31+00”,145,4,0,3,“+19703769316”,145,3
22:37:35.789 → IN : abc
22:37:35.789 → —> AT+CMGF=1
22:37:35.826 → <— OK
22:37:35.859 → —> AT+CMGD=000
22:37:35.934 → <— OK

----same 2 SMSs sent via dashboard . hologram . io/org/32579/device/637563/message

22:37:47.590 → AT+CMGR=0
22:37:47.729 → +CMGR: “REC READ”,“+1512xxxxxxx”,“”,“22/04/04,04:37:46+04”,145,4,0,4,“+447797704088”,145,1
22:37:47.805 → IN : 6
22:37:47.843 → —> AT+CMGF=1
22:37:47.843 → <— OK
22:37:47.915 → —> AT+CMGD=000
22:37:47.953 → <— OK

22:37:53.510 → AT+CMGR=0
22:37:53.652 → +CMGR: “REC READ”,“+1512xxxxxxx”,“”,“22/04/04,04:37:52+04”,145,4,0,4,“+447797704088”,145,3
22:37:53.759 → IN : 616
22:37:53.759 → —> AT+CMGF=1
22:37:53.797 → <— OK
22:37:53.835 → —> AT+CMGD=000
22:37:53.872 → <— OK

…about an hour later I tested with a longer message with less logging.

note:
1.it’s unlikely that it’s a coincidence that 61 in Hex is ‘a’

message 03: “aaaaaaaaaa”
message 04: “bbbbbbbbbb”
message 05: “cccccccccc”

----sending SMS from my phone to +882 36 00099 xxxxx

00:03:38.779 → SMS: 1
00:03:39.519 → IN : aaaaaaaaaa
00:03:57.763 → SMS: 1
00:03:58.473 → IN : bbbbbbbbbb
00:04:16.280 → SMS: 1
00:04:17.204 → IN : cccccccccc

----same 3 SMSs sent via dashboard . hologram . io/org/32579/device/637563/message

00:04:45.368 → SMS: 1
00:04:46.132 → IN : 6161616161
00:05:04.054 → SMS: 1
00:05:04.992 → IN : 6262626262
00:05:22.842 → SMS: 1
00:05:23.617 → IN : 6363636363

2

Thanks for this. This does appear to be an encoding issue on our end. We’ll investigate

1 Like
3

So for a current update this might be a problem with one of our carrier partners and their SMSC encoding formats. When I test it with the working ones I see that the messages are being sent appropriately:

AT+CMGL="ALL"


+CMGL: 1,"REC READ","+80111",,"22/04/04,17:10:39+04"
Test ascii

+CMGL: 2,"REC UNREAD","+80111",,"22/04/04,17:13:43+04"
aaaaaaaaaa

OK

However one of our partners does not seem to be handling encoding properly as you see where it is sending the messages as hex encoded.

AT+CMGL="ALL"


+CMGL: 1,"REC UNREAD","+447937405250",,"22/04/04,11:36:28-20"
74657374206173636969

OK

which is "test ascii" encoded as hex.

We will reach out to the carrier partner and try to verify things with them as the SMPP text encoding for ascii is coding 1 and it is actually dictated in the SMPP standard vs data coding 0 which is generally GSM7 though is the default of the SMSC.

4

Thank you Dom and Reuben for your quick attention. I know this isn’t Hologram and hopefully my customers know it isn’t me - with systems this massive, we’re all in it together :+1:

Please keep the updates coming as you receive them.

5

No problem, looking more into it it seems this carrier partner is also encoding data coding 8 - UCS2 (UTF-16) messages as hex encoded too so this is obviously an issue that we need to follow up on since this is non-standard behavior. We are deploying a temporary fix but UTF encoded messages might not work depending on how their SMSC is handling these messages.

We apologize for this, we are engaging with the partner to fix their encoding and are also frustrated when things don’t work as standards expect.

6

Hey @Dom, did your partner deploy a solution for this issue?

7

We created a patch for it though made some interesting observations about text encoding and modems as well.

It should work though we are not sure the full character set of ascii/gsm7 will be supported since everything is forced to use the default smsc alphabet encoding.

There does also appear to be some discrepancies between modems in that characters the modem isn’t able to support it just returns the hex encoding even though the PDU is properly encoded. For example the following PDU: 0791447779070488040C9144977304250500082240402110440A3020AC3053306B3061306F4E16754C00610041006200420063004300640044006500450066004600670047006800480069

When decoded will show that the message says: €こにちは世界aAbBcCdDeEfFgGhHi

However the modem is not able to render the euro sign and the Japanese characters so when the message format is set to 1 AT+CMGF=1 it just replies with the hex encoded message. I tested this with various uBlox modems and want to see what other brand modems support but that is the finding thus far.

I want to try other encodings that might disambiguate things such as encoding 3 which is Latin 1 ISO-8859-1 but that work is being done on a separate SMPP bind than what we use in production so will not interfere with the current working encodings.

8

Thanks @Dom, you know way more about this than I do so the thread referenced below may share nothing with this current thread except that I do still believe that character encoding issues SUCK!
All my work has been on SIMCOM SIM7000, SIM7500A and soon the SIM7600G R2. I think all the characters I use are in the ASCII set but if you need to me to test with my SIM7500A I can. It sounds like the work you’re doing is in a test env so maybe I can’t help. If you want me to try the char set you included in your last message with my modem I can fairly easily.

9

It does look like I’m seeing some stuff both my SIMCOM SIM7500A and SIM7600G_R2:
Below shows sent (using the Hologram Dashboard) and received:

!@#$%^&*() -=_+ []{} ;':" ,./<>?
!⸮#⸮%⸮&*() -=⸮+ ⸮⸮⸮⸮ ;':" ,./<>?

This isn’t a blocker for me but of course it’s not ideal.

10

Can you set your modem to get the messages in PDU mode AT+CMGF=0 and send me the PDU I want to verify that its encoding and not a modem issue

11

@Dom I sent the SIM7600G_R2 a message in text mode and read it, deleted it, changed to PDU mode, read it. Results:

string = 
!@#$%^&*() -=_+ []{} ;':" ,./<>?

09:17:35.441 -> at+cmgr=0
09:17:35.470 -> +CMGR: "REC UNREAD","+15127500974","","22/04/08,15:17:22+04"
09:17:35.591 -> !⸮#⸮%⸮&*() -=⸮+ ⸮⸮⸮⸮ ;':" ,./<>?
09:17:35.591 -> 
09:17:35.591 -> OK
09:17:49.174 -> at+cmgd=0
09:17:49.174 -> OK
09:17:53.377 -> at+cmgr=0
09:17:53.377 -> OK
09:18:12.186 -> AT+CMGF=0
09:18:12.186 -> OK
09:18:33.424 -> at+cmgr=0 
09:18:33.454 -> +CMGR: 0,"",47
09:18:33.454 -> 0791447779070488040B915121570079F40000224080518102402021E08854F29A54A814A8D5FBAE40DBEEBE0FDA9D742210CBF5E2F97E
09:18:33.566 -> 
09:18:33.566 -> OK
09:18:57.257 -> at+cmgf=1
09:18:57.257 -> OK
09:19:01.328 -> at+cmgr=0
09:19:01.368 -> +CMGR: "REC READ","+15127500974","","22/04/08,15:18:20+04"
09:19:01.417 -> !⸮#⸮%⸮&*() -=⸮+ ⸮⸮⸮⸮ ;':" ,./<>?
12

So decoding that SMS I can see that it seems your modem is unable to render the GSM7 characters it maps to which is the backwards question marks. Thing is we reverted the encoding type to its previous values so it likely was having this problem before and hence why we were trying to address the encoding in the first place.

I will keep looking into how we can disambiguate this. Its tricky since the modems themselves are also the ones decoding things and many don’t seem to be able to handle the full GSM7 character set…

13

Hey @Dom , I should have mentioned this before but things were going well and it’s low priority. You said “…Thing is we reverted the encoding type to its previous values so it likely was having this problem before and hence why we were trying to address the encoding in the first place.”

but that’s actually not true. I have to send special characters (the Hologram Device Keys in fact) to my units as part of setup and I am 100% confident that some of these special characters including the at symbol @ and curly braces { } used to work. I have record of this working with SIM7500A in the past, early/mid 2021.

14

So you are right, those characters are sendable it just depends on what other characters are sent along with it. ASCII and GSM7 renderable characters overlap except for the backtick character ` which is not present in GSM7 and will cause the encoding to instead encode things as ASCII (how we’ve implemented it) which will cause those characters to map to other characters that the modems don’t render.

Its a really weird design issue where the modems expect GSM7 encoding to parse SMS but can only render ASCII characters and sending it in the ASCII encoding makes the modem render it in hex.

I am not sure its the optimal solution but for whatever reason modem designers and the infrastructure for sending SMS don’t seem to be on the same page…

15

Ok, after reading that 4 times I think I understand. Character encoding issues are so ugly, thanks for your help and all the info.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.