Connecting NovaM to Cellular Network

Establishing a PPP Connection Using Hologram NovaM

I have been developing an IoT device based on a single-board computer (Orange Pi PC+). The device is used for vehicle tracking and includes GPS, accelerometer, gyro, and more. It sends location updates at regular intervals to a server via a cellular network. I have been using 3G GSM USB cellular modem dongles and they work fine in urban/suburban settings, but in more rural areas the 3G GSM coverage is often inadequate. Therefore, I have been seeking an LTE solution.

I have purchased a wide variety of LTE USB modems, but I have not found one that I can get to work with my device. More or less, all LTE USB modems are made in China. The market for cellular modems in the US is tiny, so the Chinese manufacturers donā€™t place much emphasis on compatibility with US carriers and, even worse, they donā€™t typically have a good way to test their devices for compatibility with US carriers. Also, they are virtually all focused on created USB modems which are Windows compatible, and know nothing about Linux. Add a language barrier, and you can understand why I have a box full of Chinese LTE USB dongles which are useless to me.

When I stumbled across the Hologram NovaM I got very excited at the prospect of an LTE modem designed in the US and supported in the US. The fact that it is open source is an added bonus.

I purchased some Hologram NovaM devices a few weeks ago and began experimenting with them using Hologram SIMs. Iā€™ve gotten further along than with any of the Chinese devices, but Iā€™m not out of the woods yet. In this document I will share what I have learned about the NovaM, exactly how I use it to establish a ppp connection, and what problems I have not yet been able to sort out. I have two hopes:

  1. this document will be helpful to others trying to use the NovaM
  2. folks at Hologram will provide further assistance

At this point, let me state clearly that I have only been experimenting with the Hologram Python tools and the NovaM for a few weeks, so I am certainly not an expert. I have posted numerous questions on community.hologram.io and have received some very helpful responses. I have also communicated by e-mai and phone with one of Hologramā€™s sales engineers who has also been helpful. Iā€™ll also add that there are some very important questions which I (and others) have asked which go unanswered. Given my understanding that Hologram developed the NovaM, I am surprised by this. The impression I get is that either nobody at Hologram is able to answer these questions, or those who can answer these questions are not aware they are being asked. Either way, without some good, basic, technical support, the NovaM cannot be adopted on large scale and certainly cannot be used except by hobbyists.

AT THE END OF THIS DOCUMENT I WILL DESCRIBE THE PROBLEMS I HAVE NOT BEEN ABLE TO SORT OUT AND FOR WHICH I HAVE NOT BEEN ABLE TO GET USEFULL ASSISTANCE FROM HOLOGRAM.

The Hologram Software

I began by using Hologramā€™s Python SDK and their CLI (Command Line Interface). The installation instructions on the Hologram web site are intended for Raspberry Pi running Raspian. I have been able to install the Python tools on my Orange Pi, and Iā€™ve documented the process. If anyone want the details, just ask.

I was able to use the CLI to connect to the Internet, but some problems arose:

  1. the tools didnā€™t always work.
  2. once a ppp connection was established, there was no way to send AT commands to the device, so there was no quick way to tell when the connection dropped. All I could do was check for the existence of a ppp0 interface. The ppp0 interface can take minutes to disappear after the connection drops.

It turns out that (2) has nothing to do with the Python tools, but without understanding the reason for (1) all I could think to do is stop using the tools and try to set up the ppp connection without them.

My current understanding is that the Python tools donā€™t do anythingā€“with regard to setting up a ppp connectionā€“beyond invoking pppd (the ppp daemon) and providing a proper chat script, so now I do that myself as I will explain.

NovaM

The NovaM is based on the,

U-blox SARA R410M-02B

module. This is an LTE CatM1 and NB-IoT device. Presently, T-Mobile is building out NB-IoT and their network is available for testing (at least). Verizon, AT&T and other network providers are offering Cat M1.

This document,

https://www.u-blox.com/sites/default/files/SARA-R4-SARA-N4_ATCommands_%28UBX-17003787%29.pdf

is full of useful information about the moduleā€™s AT command set and more.

Connecting the NovaM

Drivers are needed for the NovaM. Install them like this:

sudo  modprobe  optional
sudo  sh  -c  'echo -n 05c6 90b2 > /sys/bus/usb-serial/drrivers/option1/new_id'

After installing the drivers, inserting the NovaM into a USB port will present four devices:

/dev/ttyUSB0
/dev/ttyUSB1
/dev/ttyUSB2
/dev/ttyUSB3

Only one will respond to AT commands. It is usually,

/dev/ttyUSB2

but not necessarily. I wrote a Java program,

findmodem.jar

which attempts to open each of those devices, sends,

AT\r

and sees if it gets a response. If so, it has found the correct device. It returns the device, for example,

ā€˜/dev/ttyUSB2ā€™

or,
no_modem

if none of the devs respond.

I wrote the program in Java because I already have a JVM in my device and I could write it on my desktop computer and just copy it to my IoT device without recompiling (Java is most portable). It would be easy enough to write such a program in C.

PLEASE NOTE: the active device is usually /dev/ttyUSB2 and I believe some parts of the Python SDK hard code /dev/ttyUSB2. If it happens that the active device is not /dev/tttyUSB2 then any code expecting /dev/ttyUSB2 will, obviously, not work.

Establishing a PPP Connection

To establish a ppp connection:

  1. create a chat script for the ppp daemon (pppd)
  2. configure pppd
  3. enable pppd logging
  4. invoke pppd

Step 1. Create a Chat Script

Create a file called,

chat_nova

containing (without the indenting):

ABORT 'BUSY'
ABORT 'NO CARRIER'
ABORT 'VOICE'
ABORT 'NO DIALTONE'
ABORT 'NO DIAL TONE'
ABORT 'NO ANSWER'
ABORT 'DELAYED'
TIMEOUT 12
REPORT CONNECT

"" AT
OK ATH
OK ATZ
OK ATQ0
OK ATD*99***1#
CONNECT ''

You can store the file wherever you want. I stash it at:

/etc/ppp/chatscripts/chat_nova

I got this chat script from the Hologram github site, so no credit to me for this.

Step 2. Configure pppd

To configure the ppp daemon create (or edit) the file,

/etc/ppp/options

to contain (without the indenting):

/dev/ttyUSB2
115200
noipdefault
usepeerdns
defaultroute
persist
noauth
connect "/usr/sbin/chat -v -f /etc/ppp/chatscripts/chat_nova"

Notes:

  1. the first line contains the active NovaM device. It will be wise to dynamically set the correct value just before Step 3. I do this by using the sed utility.

  2. the last line invokes chat and specifies the chat script to use (created in Step 1)

(Again, I got these options from the Hologram GitHub site, so no credit to me.)

Step 3. Enable pppd Logging

In the absence of some way to monitor what pppd is doing, the connection will succeed or fail but there will be no clues as to why. Enabling pppd logging will cause pppd to log its activity, and weā€™ll be able to watch what pppd is doing in real time.

To enable pppd logging,

  1. Edit,

    /etc/ppp/options

and add these two lines:

debug
kdebug 7
  1. Edit,

    /etc/rysyslog.d/50-default.conf

and uncomment these lines:

*..=debug;...
*.=debug;\
auth,authpriv.none;\
news.none;mail.none	-/var/log/messages

Now, pppd will write debug information to:

/varr/log/debug

To watch the log as pppd is running, open a console and run tail:

sudo  tail  -f  /var/log/debug

Step 4. Invoke pppd

Now, run pppd,

sudo /usr/sbin/pppd

If all goes well pppd it will establish a ppp connection. Once a connection is established running,

sudo  ifconfig

will show a ppp0 section which will contain the IP address of the device.

Remaining Problems

  1. Canā€™t Communicate with NovaM after PPP connection is established

On page 11 of the U-Blox document (the link appears early in the document) three modes are described:

Command Mode: there is no connection and the device responds to AT commands

Data Mode: a connection has been established and device does not respond to AT commands

Online Command Mode: a connection has been established and the device responds to AT commands

The documentation indicates that it is possible to switch from Command Mode to Online Command Mode. I have not been able to accomplish this, and I have not been able to obtain any useful assistance from Hologram.

This is a significant problem, but much less so than the next one.

  1. No Network Protocols Running

Sometimes when attempting to establish a ppp connection all goes well. Sometimes it does not. When it does not, it is quite common to see a message in the log containing this text:

No network protocols running

I have spent quite a few hours trying to get to the bottom of this one. I know Iā€™m not the only person running into this problem because,

a. Iā€™m just not that special
b. I believe others have reported this on the community formum

  1. It Can Take 20 Seconds or 5 Hours For the Red LED to Light

I have not yet identified rhyme or reason as to why the NovaM, when powered on, can take seconds or hours for the Red LED to light. Iā€™m formulating some theories, but I donā€™t yet have enough experience to take any position. Many, many, many others have raised this problem on the forum and Hologram doesnā€™t seem to have any useful guidance. Is it firmware problems in the U-Blox module? Is it a provider/Hologram SIM problem? Is it something else?

A Plea for Help

If I were using a Hologram modem and non-Hologram SIMs, or vice-versa, I would not necessarily expect Hologram to get to the bottom of any of these problems, but Iā€™m using a NovaM and Hologram SIM cards.

It would be very nice if Hologram provided some useful technical support or explained why they cannot. Perhaps the NovaM design was outsourced and Hologram really doesnā€™t have folks with the necessary technical knowledge to help.

Iā€™ll add that I have purchased 56 NovaM devices. My intention was to deploy 50 of them with one of my new customers and keep the rest for field testing. Once I find a suitable device, I will need about 1,000/year. Referring only to my actual purchase of 56 devices, I donā€™t know if that makes me a ā€œbigā€ user or a ā€œsmallā€ one, but I feel that with such a purchase Hologram should provide more technical support than suggesting I search the Internet or post on the community forum. One reason I was attracted to Hologram was the prospect of purchasing devices and connectivity from a single source to avoid finger-pointing. It didnā€™t occur to me that instead of finger-pointing I would just be, more or less, ignored.

Finally, let me say that I am NOT blaming @Reuben or @Maiky or anyone else who contributes to this forum. I very much appreciate the help I have gotten on the forum. Rather, my point is that Hologram is commercial venture (nothing wrong with that). They do not give away connectivity (beyond tasting) or hardware and that is precisely why I want to use their services and products: for profit companies can and should provide solid support for their products.

Regards,
Matt

4 Likes

Iā€™ll take a closer look at your problems and see if I can help tomrrow but some quick thoughts especially as you look towards scaling out to 1000+ units / year.

At that volume, I wouldnā€™t use an OrangePi PC + Nova, you have enough scale to build a custom PCB and solder on a SARA-R410M yourself. If you are just telemetering relatively low data rate information (on the order of 10kB/hr or less) I would suggest ditching linux and using Arduino or other lower level embedded architectures. You are over-paying for hobbyist tools (both the Nova and the Orange Pi). THe downside is R&D development and circuit design but you can probably cut your hardware costs in 1/2 if not more.

Also if you go the path I metion above you dont bother fighting linux and Hologramā€™s SDK you do everything with AT commands. You can still interface with Holgorams cloud and you have more direct control over the hardware and ability to change the imlementation. Again this is lower level programming, may be harder R&D up front but will pay off down the road.

This is the path I took using a microcontroller with Arduino compatibility and interfacing with a U-blox modem directly through AT commands on my own PCBA. At 1,000 units my cost of PCB + assembly + Wifi chip, R410M, simcard, microcontroller, antenna, sensors is just over 1/2 the cost of a Nova.

Hi Andrew, and thanks for your reply.

The solution I have developed is probably beyond the capacity of an Arduino. My device, called Spotter, includes an Orange Pi (OPi) attached to a custom circuit board with power supply, power management, audio amplifier (for an external speaker), cooling fan support, 3-axis acc/gyro, diagnostic LEDs, 4 analog inputs, and routing (to the OPi) for 2 RS-232 connectors. The OPi includes 8gb of flash too, so I donā€™t need to use (expensive and unreliable) SD cards.

Iā€™m using Linux because Spotter performs text-to-speech (TTS) to interact with the school bus driver. Spotter also includes geo-spatial data (map data) so it can speak turn-by-turn directions. 99% of the firmware is written in Java (using JNI to control and interact with gyro, accelerometer, power management PIC) where multi-threading makes it very easy to create separate thread for turn-detection, speak, handle keyboard, control LEDs, etc., etc.

Iā€™m not sure I would save money by avoiding the OPi. It costs about $23 (delivered) and in qty 1,000 I donā€™t think I could buy the equivalent parts for that prices. Iā€™ve had some discussions with the manufacturer of the OPi and at qty 2,000 they are willing to make a custom run leaving off some hardware I donā€™t need (e.g. ethernet and IR) to lower the cost a bit. Also, using the OPi give me access to Armbian/Ubuntu which is pretty well maintained, so I avoid all the problems of porting Linux to a custom board. And, of course, Java is happy on Linux.

Youā€™re definitely right about saving money on the ublox module. Paying $50 for a NovaM is a lot. Iā€™m certainly considering using the module in my custom board. At present, however, I have immediate need for an LTE solution. If I can get the NovaM working reliably that will solve my immediate problem and prove that the R410 module is one I can use.

Again, thanks for you help.

Hey @mbrenner thanks for the detailed post here. We have been taking a look at your issues.
Hereā€™s what I can respond with right now:
Issue 1: We havenā€™t used this feature yet but are reaching out to UBlox for more information.
Issue 2: I actually havenā€™t seen that one. Can you post the log with some more context?
For issue 3, this can usually be solved by turning off NB, doing a COPS=? network scan or picking specific network profiles with UMNOPROF. (See the main R410 thread)
Weā€™re also working with carriers on solving some timing issues that could make things run a little smoother here.

This kind of brings me to your overall point about the process. Cat-M1 is still in early days with few larger deployments and many individual carriers (as well as the whole roaming system) still shaking out bugs and dealing with issues. Many of the new features of M1 (like longer radio idle power down times for example) actually cause problems with older systems and require updates.

The R410 module has its own issues partly due to M1 and partly due to the whole platform being a significant departure from the previous ā€œUā€ series of modules.

We definitely understand the frustration here and weā€™re doing everything we can to make the process better and are in continuous conversations with carriers and with UBlox to try to get improvements on various systems. It may just take some patience while these things are improved.

Thanks for reaching out and weā€™ll keep trying to assist wherever possible.

Ah gotcha, yea you are doing much more than I originally thought with the TTS and turn-by-turn. Although it may be possible on an esp-32 ($3.80 with 2.4G wifi + bluetooth) with an external audio front-end. see: Alexa on ESP32 | Espressif Systems Not saying you should go this route, but just mentioning there are Arduino compatible MCUā€™s with some decent performance.

I see they have an OPi 4G-IOT product, very limited documentation on it from what I can see (doesnā€™t say if its LTE-1, M1, NB-IOT). It looks like they use SKYWORKS cellular front-ends on the boards (at least their 3g product does) and SKYWORKS has M1/NB-IOT chips so its possible the OPi 4G-IOT might be another route for you. It potentially keeps everything on one board and ensures compatibility between the computer and modem. Also gets everything on one board for you. You may have looked at this already but I thought it was worth mentioning.

I definitely have issues with initial connection with the R410M. I went the route of setting the UMNOPROF and trying to connect for a brief period (say 2-5 minutes) and then cycling through other common carriers. For example I go Verizon ā†’ AT&T ā†’ Telstra ā†’ T-Mobile, repeat doubling dwell time at each profile. My product is only in USA and Australia so I only hunt there. Also to improve re-connect speed, I store the profile in non-volatile memory where I did successfully connect and try that network first (this way the worst hunting should only occur once).

I cant help with the PPP and commanding as I solely use AT commands and have relatively low bandwidth and no real latency concerns. It may be worth reaching out to U-Blox directly as well, the Nova is like 95% U-Blox Modem and the issues you are having with PPP and Online command mode are really features of a U-blox product. If nothing else, it might help having both Hologram and you going to U-Blox with similar issues to get faster responses from U-blox.

Hey, thanks very much for pointing me to the esp-32 and the OPi 4G-IOT. Iā€™ll take a close look at each. Even if they arenā€™t ideal (maybe they are) itā€™s always interesting to know whatā€™s out there.

Iā€™ve finally got things working pretty well connection-wise, though I have to do more testing on the road (literally) to see what happens when the NovaM is moving between and across cells. My main goal is to minimize time without a connection because parents and students are using mobile apps (part of my product offering) to track the school buses they ride and cellular dead zones of more than a minute or two are noticeable. This is mostly a concern in more rural areas where the coverage can be incomplete.

Iā€™ve also experimented with using UMNOPROF to stick with a single provider as I believe Verizon may have the best coverage in rural areas (since they have the most towers).

You write that you donā€™t use PPP but, rather, do everything yourself with AT commands. This is very interesting to me. Are you able to open a socket connection and send a packet just using AT commands? If so, this would avoid the problem of losing access to the the port (no AT commands) while the PPP connection is active.

Thanks for sharing your ideas and experience.

Regards,
Matt

FYI, that AT method is how the hologram SDK sends data to our cloud when you do hologram send whatever. We donā€™t open a PPP session either and open a socket using AT+USOCR and related commands.

1 Like

We only use PPP for the hologram network connect command

Yep thats exactly what I do, something like this (just example only, probably will need some minor changes to actually work):
AT+USOCR=6 << create a socket (TCP, returns socket number, assume ā€œ1ā€)
AT+USOCO=1,"www.google.com",80,0 <<connect to server www.google.com on port 80
AT+USOWR=1,12,"Hello world!" <<send data to server
AT+USORD,1,100 <<read up to 100 bytes of response data from server
AT+USOCL=1 <<close socket

If you are sending larger amounts of data, like Audio files, you may want to look at the AT+USODL command which kind of works like PPP where you set the link to direct link mode where whatever you send on the COM port is sent to the server and whatever the server response with is sent back to you on the COM port. You escape this by sending ā€˜+++ā€™ followed by 2 seconds of silence. I havenā€™t tested this as USOWR is sufficient for me.

As I mentioned above, I like this way as I have more control as I can check network status, signal strength, etc before attempting to connect and potentially get more information on why communication failed (did it fail to connect to server port? succeed in connecting but failed to send data? did the remote server close the port prematurely?)

This works better for smaller packets of data without tight latency controls as you have the timing overhead of AT commands and the module buffering data instead of sending it directly to you.

Checkout section 18 in the AT command manual for details.

2 Likes

oops, clicked the wrong reply buttonā€¦

Well, thanks for the rough answer to how to send packet using AT commands. I think that will serve well for my needs. I send location updates (lat, lon, UTC, speed, and some status bits in 17 byte packets.

The problem I have using a ppp connection is that once established I can no longer send AT commands to the Nova. Using AT commands to send the packet will allow me to stay in Command mode where I can monitor signal strength, etc.

You mention at the end of your post, ā€œā€¦timing overhead of AT commands and the module buffering data instead of sending it directly to you.ā€

Can you expand on, ā€œā€¦the module buffering dataā€¦ā€? I ask because Iā€™ve noticed some unexpected buffering. Hereā€™s the scenarioā€¦

I establish a connection and my tracker, using the NovaM, is sending location information every 30 seconds. For testing purposes, I want to simulate driving into a cellular dead zone so I disconnect the antenna and I can see that my server stop receiving packets. After a few minutes I reconnect the antenna and shortly thereafter all of the missing packets arrive in quick succession.

I know that my tracker was not buffering the packets. It WOULD buffer the packets if it could have determined that the connection was down, but it had no such indication. This means NovaM (Ublox module) must be storing them.

It would be nice to know exactly what is going on. That is, under what circumstances does the module buffer? How large is the buffer? Under what conditions will it discard the buffer? Is it possible to clear the buffer? etc.

Cheers,
Matt

@mbrenner Just to jump back to your previous question, we heard back from UBlox about the Online Command Mode that you were asking about.
Itā€™s actually the same thing that @AndrewGifft mentioned above about sending +++ and getting back to an AT mode.
After sending the ATD dial string that you posted above and getting your CONNECT response you can send +++ and wait a couple seconds and youā€™ll get back to being able to send AT commands.
I just confirmed this works on my modem but you have to make sure you are sending at the correct baud rate, 115200, when youā€™re sending the +++ string.

What I was alluding to with buffering
With AT commands you dont get any received data directly to you it will send a ā€œunsolicited result codeā€ (URC) that says data is available. Until you actually request the data via AT+USORD it buffers data it received. Similar with sending data, you tell the modem via the 2nd parameter to AT+USOWR how much data you are going to send. Depending on the internal implementation, it may buffer your data locally until it receives the full payload and then begin sending data packets over TCP. With PPP it may behave differently as it does not know the full payload size. Basically I could see the sum of all these delays for send and receive being like 400-1000ms on top of network latency. Not bad for most applications but if you needed to get as close to realtime as possible this may be unacceptable.

Your experience with buffering
Check out the very bottom of page 195, top of 196 in the at command manual: https://www.u-blox.com/sites/default/files/SARA-R4-SARA-N4_ATCommands_(UBX-17003787).pdf

If no network signal is available, the TCP packets are enqueued until the network will become available again. If the TCP queue is full the +USOWR command will return an error result code. To get the last socket error use the +USOCTL=1 command. If the error code returned is 11, it means that the queue is full.

I believe this is the behavior you saw. The modem keeps the socket open until explicitly closed by you or explicitly closed by the server, or likely some long timeout (few minutes+ I would guess). This means when you pull the antenna, the socket remains in the connected state so the module will buffer the data locally. EDIT: Note you can check the status of your socket via the AT+USOCTL=#,10 command where # is the socket number

I dont see explicit answers in the AT command manual for your following questions but my guesses and rationale are below:

under what circumstances does the module buffer? When a TCP connection is open but no network connectivity is available (per AT command manual).

How large is the buffer? I would guess enough for a few kB of data, like 8kB+. You can test this by trying to send larger and larger packets with the antenna removed but TCP established.

Under what conditions will it discard the buffer? I would guess only when the connection is closed either via AT command, explicit close connection command from the server, or the many minute+ timeout I assume exists.

Is it possible to clear the buffer Best way would be to close the socket via AT command and you can immediately re-open it to try again without a full buffer.

1 Like

Hi Reuben, thanks very much for following up. Let me share my thinking given the information you sharedā€¦

Under Linux, it is usual to launch the PPP daemon (/usr/sbin/pppd) to create a ppp connection. In the configuration file for pppd (/etc/ppp/options) a ā€œchat scriptā€ file is typically provided. pppd uses the chat program to ā€œchatā€ with the modem (send it commands and handle responses). The chat script contains the AT commands (including ATDā€¦) which should be sent to the modem.

So, the problem is that pppd opens the dev attached to the modem (e.g. /dev/ttyUSB2). Since pppd opens the dev, it cannot simultaneously be opened by another program. So, it seems like there are two possible ways to send the escape sequence (+++) to the NovaM:

  1. modify pppd, writing in a hook, to enable it to send +++ to perform the switch, another hook to allow interaction with the modem after the +++, and a third hook to switch back to Data Mode. While this is theoretically possible, it is a daunting task and itā€™s probably not a great idea to create a variant of a basic utility.

  2. donā€™t use pppd. Instead use AT commands to do everything (as @AndrewGifft described). The dev must be opened to send the AT commands so, of course, +++ can be easily sent to the opened dev.

Does it seem to you that Iā€™ve got this right?

You know, I started off using the Hologram SDK and CLI. I backed off that to set up my own ppp connection. Now Iā€™m backing off that to simply control the NovaM with AT commands. Pretty soon Iā€™ll back off the Nova and my tracker altogether and just have the bus driver yell his location out the window. :wink:

Cheers,
Matt

1 Like

Am I correct to assume that once you establish the ppp connection you have no way to switch from Data Mode to Online Command Mode?

Communicating with U-Blox is certainly a good idea, but they wonā€™t give me the time of day.

Okay, after sleeping on the idea of handling all data communication myself using AT commands, I realized the biggest drawback: my firmware is multi-threaded with one thread sending location data, another periodically checking for software updates, another requesting directions when needed, another periodically sending driver hours, etc., etc.

There is no problem letting each thread do itā€™s job when there is a ppp connection. Linux uses the connection and takes care of all the details. Avoiding ppp and writing a solution to handle each threads needs is a complex design matter which will likely include bugs which will be devilishly difficult to sort out. Also, as the devices are deployed all over the place, updating for bug fixes will not be much fun.

Also, as I think about it, the main challenge is not to able to send AT commands whenever I want (thought that is good too), but rather to determineā€“as soon as possibleā€“that the cell connection has dropped so the firmware can decide what to do.

The main wrinkle has been that Iā€™ve been sending location data via UDP. My thinking has been that since my data payload in a location packet is only 17 bytes, the additional overhead of the TCP packet header (bigger than a UDP header) and the TCP handshaking will increase my data consumption by more than 100%. Also, my experience, after sending millions of UDP packets is that a small fraction of 1% of packets get lost (I cache packets while there is no connection and send them when the connection is regained).

What Iā€™m finding with the Nova is that it is very difficult, once a ppp connection is established (and I can no longer send AT commands), to determine when the ppp connection drops. Running ifconfig will show ppp0 long after the connection is gone (5+ minutes). Checking /proc/net/dev is similar (I think ifconfig gets its info from /proc/net/dev).

The discussion of buffering seems to indicate that the R410 module even keeps it a ā€œsecretā€ that it has lost the connection and buffers data hoping the connection will return. Given the desire of the module to ā€œhideā€ the dropped connection, I think no error occurs when a UDP packet canā€™t be sent to the carrier and, of course, once a carrier sends a UPD packet there is never any kind of acknowledgement of receipt.

All of this suggests that using this module Iā€™m not going to be able to send UDP packets and know, in a timely fashion, that the connection has dropped.

So, my latest thought is to use TCP to send location information. If I donā€™t get a prompt acknowledgement (within, say, 10 secs) then I can interpret this to mean the network connection was lost and the last packet sent should be cached for later sending.

Do you have experience with this? When you send a TCP packet do you get accurate acknowledgement? That is, if there is no error, does that mean it was delivered to the network and a response returned by the destination server, or is the module giving a false ACK while it buffers the data for later sending?

If the module is sending false ACKs then of course, I can have my server send back a byte of data indicating success and then Iā€™ll know for sure, but if I donā€™t need for the server to send back info then why waste the dataā€¦

Obviously, I need to perform my own tests, but if you have any experience to share Iā€™ll be most grateful.

-Matt

Yea at the end of the day, you have to change something about your implementation. It can be either further debugging ppp, trying to work with U-Blox to get online command mode and switching states to work, it can be writing a thread-safe wrapper around your COM port to send / receive data via AT commands for multiple threads, or it can be single threading your application and doing event based or simple periodically checking of states. Unfortunately ā€œnone of the aboveā€ doesnā€™t seem like an option.

Depending on your comfort level with writing thread-safe applications writing a wrapper over around the COM port would allow you to keep most of your software in place.

Or since all of your events are relatively slow for computers (as they are driven by human input and vehicle location) you could just have a master thread or everything in one thread and just check states of everything at like 20Hz. No one will notice if a command to turn is 50ms later, or if you have 50ms jitter on your location positioning.

Lastly, although it is somewhat annoying, I see why the U-Blox device implements buffering. Cellular networks, and all wireless networks are somewhat unreliable, in most cases, if you loose signal for 2s and then get it back, you would want everything to just resume gracefully as if nothing happened. I would rather a web page just takes an extra second to load than to error out, similarly I would rather a phone call just have 1s of silence and resume then to drop the call just because the network went out briefly. Now maybe there should be a configurable timeout, like ā€œif I am trying to get online for 30s and still nothing, then give me an errorā€.

Answers to some of your questions

Question All of this suggests that using this module Iā€™m not going to be able to send UDP packets and know, in a timely fashion, that the connection has dropped.

Answer You can do this through AT commands to atleast check network registration state just prior to sending the UDP packet. Especially useful if you use AT commands for everything (including sending data)

Question When you send a TCP packet do you get accurate acknowledgement? That is, if there is no error, does that mean it was delivered to the network and a response returned by the destination server, or is the module giving a false ACK while it buffers the data for later sending?

Answer See AT command manual middle of page 196 for AT+USOWR:

The information text response indicates that data has been sent to lower level of protocol stack. This is not an indication of an acknowledgement received by the remote server the socket is connected to.

Basically when you get OK response to AT+USOWR... command it means the modem accepted your data, not that it has been successfully sent and received by the server. I donā€™t think the socket will give a false ACK and you can check the status of the TCP connection with the AT+USOCTL command.

Question If the module is sending false ACKs then of course, I can have my server send back a byte of data indicating success and then Iā€™ll know for sure, but if I donā€™t need for the server to send back info then why waste the data

Answer This may not be required if the result of AT+USOCTL is sufficient for you, that being said, I would have your server respond, it is an easier way to know at the application layer that everything went through correctly.

Additional Thoughts

  1. TCP does have a decent amount of overhead especially for small payload sizes, what confuses me though is that you must have a lot more than this going on right? You said you have multiple threads running in parallel checking for updates, sending this lat/long/speed/etc data, gathering turn-by-turn directions (Iā€™m guessing you do this via some google API), speech to text (Are you doing this all on the device or using some cloud service like Alexa). If so its seems this overhead on only one of the types of communication going over this link is unnecessary optimization. Also your data consumption must be relatively high anyway if all this is going on.
  2. If you control the server for these messages, which it sounds like you do, and you are sending data quickly you may be able to keep the TCP connection open on both sides, atleast reducing the relatively high data cost of establishing and gracefully closing a connection.
  3. Iā€™m not sure why it is important to buffer data and detect if the network is out. For your use case what seems to matter is ā€œwhere is the bus nowā€ not ā€œwhere was the bus beforeā€. So the only bit of data that is important is the most recent right? If you donā€™t have network connectivity the data collected during that time is not valuable as it will be stale by the time the network comes back up?
  4. Lastly, although its long and boring, I would suggest reading all of section 7, 13, 17, and 18 of the AT command manual. It really helps to understand what you can and cannot do with the modem related to network communication.

Again, thank you very much for your thoughtful and informative reply.

Let me respond first to some of your ā€œAdditional Thoughtsā€ as they will provide some project context.

Additional Thought 1

My tracker sends location info every 30 seconds while the bus is running (it can detect whether or not the bus running). I send the locations via UDP to minimize data consumption.

Turn-by-turn directions are done entirely on the device. Text-to-speech is done entirely on the device. Once a day (or so) the device checks in with the server to see if an update is available. Usually there is not, so thatā€™s one small packet per day. Once in a while the tracker checks with the server to see if the route the bus runs has changed. Again, usually not, so one packet. I use UDP for the location packets and TCP for everything else.

On a typical school bus, my tracker consumes about 1mb of data/month. Switching from UDP to TCP will, I expect, increase my data consumption by 100% or so.

Additional Thought 2

I send locations every 30 seconds or so. With UDP, I wrote a simple server which listens for incomming packets, plucks the data from the UDP packets and waits for the next one. With TCP, keeping the connection open shouldnā€™t be difficult, but there can be many thousands of buses so I have to think through the consequence consequences holding all the connections open for 6+ hours per day.

Additional Thought 3

Yes, it would seem that only the current location matters, but thatā€™s actually the case. For a parent or student tracking her bus, only the current location matters. But my customer (the one who pays) is actually the school districtā€™s transportation department. A key feature for them is that the system stores all location data for the entire school year. The system produces graphical traces (markers on a map) of the historical path of any bus during any period. It can produce a wide variety of reports (miles/hours operated for each bus, excessive idling, speeding, etc.) all by analyzing the historical location information. It can even determine where the actual bus stops are located and the details of the bus route, again based on the historical location info. So, losing on occasional location packet is no big deal, but losing more than that can interfere with the quality of the reporting and analysis.

Multiple AT Command Interfaces

In this document:

https://www.u-blox.com/sites/default/files/SARA-R4-SARA-N4_ATCommands_%28UBX-17003787%29.pdf

Section ā€œB.5 Multiple AT command interfacesā€ (p. 309) indicates that more than one program can access the deviceā€™s AT commands. It is a bit terse and, of course, there is no example. Have you come across this? Do you know anything about it?

An Entirely Different Solution

There is another possible solution, which you raised early on: put a U-Blox (or other) module directly on my custom circuit board. On p. 11 of the same document, it says:

SARA-R4 / SARA-N4
u-blox cellular modules can implement more than one interface between the DTE and the DCE, either
virtual interfaces (multiplexer channels) or physical interfaces (UART, USB, SPI, etc., when available).

The Nova device does not expose additional physical interfaces, but in a custom design one could. Do you know if what they are saying is that, for example, using UART and USB interfaces it is possible to access the module from two separate programs?

Again, thanks for your thoughts and suggestions.

Ok I think I understand your system better. It sounds like 99.5% of the time only one ā€œthreadā€ or application needs to be on the network, this is your 30s interval of location information. Then 1x per day you might need to download updates via one or a few files for your updates. How long would this take, 10-60s? Also ā€œonce in a whileā€ (like 2-5x per day?) you need to check for route updates. I am betting this would take less time to download, like 5-30s. It seems like synchronization between these tasks would be pretty easy. Also it seems network communication isnt really critical (in the sense the bus driver wont miss a stop, or miss a turn if you cant send/receive data).

Why not have one thread that does all your necessary communication? One dumb way to do this could be to have your gps thread just dump location data every 10-15s to a file like YYYY_MM_DD_HH_mm_ss_latlng.json, then every 30s your data communication thread looks if any files exist and sends their contents to your server. Similarly if a file ā€œDOWNLOAD_UPDATE.txtā€ exists your code will check and download an update and store the new files in a certain location. Lastly if ā€œCHECK_ROUTE.txtā€ exists, it will try to download a new route and store the new route file somewhere pre-defined. Now there are smarter ways to signal between threads / processes, local sockets being an easy one but you get the idea. If my timing estimates are correct, at worst you will have 60s between location messages to your server vs 30s, you could also build in some smarts such as you only ā€œCHECK_ROUTEā€ or ā€œDOWNLOAD_UPDATEā€ if the bus isnt on or if no latlng data is present.

Honestly if I was building this product, thatā€™s how I would implement it, you donā€™t really need parallel network access.

As for your thought about using multiple interfaces to the device. This could work, I have never done it, and there are warnings about sending conflicting settings on multiple command ports at the same time. So you will likely need one ā€œmasterā€ controller that does things like initialize and setup the modem and ensure your other ports are only used for data communication.