Connecting NovaM to Cellular Network

Hardware (and firmware) Nova
3

Hi Andrew, and thanks for your reply.

The solution I have developed is probably beyond the capacity of an Arduino. My device, called Spotter, includes an Orange Pi (OPi) attached to a custom circuit board with power supply, power management, audio amplifier (for an external speaker), cooling fan support, 3-axis acc/gyro, diagnostic LEDs, 4 analog inputs, and routing (to the OPi) for 2 RS-232 connectors. The OPi includes 8gb of flash too, so I don’t need to use (expensive and unreliable) SD cards.

I’m using Linux because Spotter performs text-to-speech (TTS) to interact with the school bus driver. Spotter also includes geo-spatial data (map data) so it can speak turn-by-turn directions. 99% of the firmware is written in Java (using JNI to control and interact with gyro, accelerometer, power management PIC) where multi-threading makes it very easy to create separate thread for turn-detection, speak, handle keyboard, control LEDs, etc., etc.

I’m not sure I would save money by avoiding the OPi. It costs about $23 (delivered) and in qty 1,000 I don’t think I could buy the equivalent parts for that prices. I’ve had some discussions with the manufacturer of the OPi and at qty 2,000 they are willing to make a custom run leaving off some hardware I don’t need (e.g. ethernet and IR) to lower the cost a bit. Also, using the OPi give me access to Armbian/Ubuntu which is pretty well maintained, so I avoid all the problems of porting Linux to a custom board. And, of course, Java is happy on Linux.

You’re definitely right about saving money on the ublox module. Paying $50 for a NovaM is a lot. I’m certainly considering using the module in my custom board. At present, however, I have immediate need for an LTE solution. If I can get the NovaM working reliably that will solve my immediate problem and prove that the R410 module is one I can use.

Again, thanks for you help.

5

Hey @mbrenner thanks for the detailed post here. We have been taking a look at your issues.
Here’s what I can respond with right now:
Issue 1: We haven’t used this feature yet but are reaching out to UBlox for more information.
Issue 2: I actually haven’t seen that one. Can you post the log with some more context?
For issue 3, this can usually be solved by turning off NB, doing a COPS=? network scan or picking specific network profiles with UMNOPROF. (See the main R410 thread)
We’re also working with carriers on solving some timing issues that could make things run a little smoother here.

This kind of brings me to your overall point about the process. Cat-M1 is still in early days with few larger deployments and many individual carriers (as well as the whole roaming system) still shaking out bugs and dealing with issues. Many of the new features of M1 (like longer radio idle power down times for example) actually cause problems with older systems and require updates.

The R410 module has its own issues partly due to M1 and partly due to the whole platform being a significant departure from the previous “U” series of modules.

We definitely understand the frustration here and we’re doing everything we can to make the process better and are in continuous conversations with carriers and with UBlox to try to get improvements on various systems. It may just take some patience while these things are improved.

Thanks for reaching out and we’ll keep trying to assist wherever possible.

6

Ah gotcha, yea you are doing much more than I originally thought with the TTS and turn-by-turn. Although it may be possible on an esp-32 ($3.80 with 2.4G wifi + bluetooth) with an external audio front-end. see: Alexa on ESP32 | Espressif Systems Not saying you should go this route, but just mentioning there are Arduino compatible MCU’s with some decent performance.

I see they have an OPi 4G-IOT product, very limited documentation on it from what I can see (doesn’t say if its LTE-1, M1, NB-IOT). It looks like they use SKYWORKS cellular front-ends on the boards (at least their 3g product does) and SKYWORKS has M1/NB-IOT chips so its possible the OPi 4G-IOT might be another route for you. It potentially keeps everything on one board and ensures compatibility between the computer and modem. Also gets everything on one board for you. You may have looked at this already but I thought it was worth mentioning.

I definitely have issues with initial connection with the R410M. I went the route of setting the UMNOPROF and trying to connect for a brief period (say 2-5 minutes) and then cycling through other common carriers. For example I go Verizon → AT&T → Telstra → T-Mobile, repeat doubling dwell time at each profile. My product is only in USA and Australia so I only hunt there. Also to improve re-connect speed, I store the profile in non-volatile memory where I did successfully connect and try that network first (this way the worst hunting should only occur once).

I cant help with the PPP and commanding as I solely use AT commands and have relatively low bandwidth and no real latency concerns. It may be worth reaching out to U-Blox directly as well, the Nova is like 95% U-Blox Modem and the issues you are having with PPP and Online command mode are really features of a U-blox product. If nothing else, it might help having both Hologram and you going to U-Blox with similar issues to get faster responses from U-blox.

7

Hey, thanks very much for pointing me to the esp-32 and the OPi 4G-IOT. I’ll take a close look at each. Even if they aren’t ideal (maybe they are) it’s always interesting to know what’s out there.

I’ve finally got things working pretty well connection-wise, though I have to do more testing on the road (literally) to see what happens when the NovaM is moving between and across cells. My main goal is to minimize time without a connection because parents and students are using mobile apps (part of my product offering) to track the school buses they ride and cellular dead zones of more than a minute or two are noticeable. This is mostly a concern in more rural areas where the coverage can be incomplete.

I’ve also experimented with using UMNOPROF to stick with a single provider as I believe Verizon may have the best coverage in rural areas (since they have the most towers).

You write that you don’t use PPP but, rather, do everything yourself with AT commands. This is very interesting to me. Are you able to open a socket connection and send a packet just using AT commands? If so, this would avoid the problem of losing access to the the port (no AT commands) while the PPP connection is active.

Thanks for sharing your ideas and experience.

Regards,
Matt

8

FYI, that AT method is how the hologram SDK sends data to our cloud when you do hologram send whatever. We don’t open a PPP session either and open a socket using AT+USOCR and related commands.

1 Like
9

We only use PPP for the hologram network connect command

10

Yep thats exactly what I do, something like this (just example only, probably will need some minor changes to actually work):
AT+USOCR=6 << create a socket (TCP, returns socket number, assume “1”)
AT+USOCO=1,"www.google.com",80,0 <<connect to server www.google.com on port 80
AT+USOWR=1,12,"Hello world!" <<send data to server
AT+USORD,1,100 <<read up to 100 bytes of response data from server
AT+USOCL=1 <<close socket

If you are sending larger amounts of data, like Audio files, you may want to look at the AT+USODL command which kind of works like PPP where you set the link to direct link mode where whatever you send on the COM port is sent to the server and whatever the server response with is sent back to you on the COM port. You escape this by sending ‘+++’ followed by 2 seconds of silence. I haven’t tested this as USOWR is sufficient for me.

As I mentioned above, I like this way as I have more control as I can check network status, signal strength, etc before attempting to connect and potentially get more information on why communication failed (did it fail to connect to server port? succeed in connecting but failed to send data? did the remote server close the port prematurely?)

This works better for smaller packets of data without tight latency controls as you have the timing overhead of AT commands and the module buffering data instead of sending it directly to you.

Checkout section 18 in the AT command manual for details.

2 Likes
11

oops, clicked the wrong reply button…

12

Well, thanks for the rough answer to how to send packet using AT commands. I think that will serve well for my needs. I send location updates (lat, lon, UTC, speed, and some status bits in 17 byte packets.

The problem I have using a ppp connection is that once established I can no longer send AT commands to the Nova. Using AT commands to send the packet will allow me to stay in Command mode where I can monitor signal strength, etc.

You mention at the end of your post, “…timing overhead of AT commands and the module buffering data instead of sending it directly to you.”

Can you expand on, “…the module buffering data…”? I ask because I’ve noticed some unexpected buffering. Here’s the scenario…

I establish a connection and my tracker, using the NovaM, is sending location information every 30 seconds. For testing purposes, I want to simulate driving into a cellular dead zone so I disconnect the antenna and I can see that my server stop receiving packets. After a few minutes I reconnect the antenna and shortly thereafter all of the missing packets arrive in quick succession.

I know that my tracker was not buffering the packets. It WOULD buffer the packets if it could have determined that the connection was down, but it had no such indication. This means NovaM (Ublox module) must be storing them.

It would be nice to know exactly what is going on. That is, under what circumstances does the module buffer? How large is the buffer? Under what conditions will it discard the buffer? Is it possible to clear the buffer? etc.

Cheers,
Matt

13

@mbrenner Just to jump back to your previous question, we heard back from UBlox about the Online Command Mode that you were asking about.
It’s actually the same thing that @AndrewGifft mentioned above about sending +++ and getting back to an AT mode.
After sending the ATD dial string that you posted above and getting your CONNECT response you can send +++ and wait a couple seconds and you’ll get back to being able to send AT commands.
I just confirmed this works on my modem but you have to make sure you are sending at the correct baud rate, 115200, when you’re sending the +++ string.

14

What I was alluding to with buffering
With AT commands you dont get any received data directly to you it will send a “unsolicited result code” (URC) that says data is available. Until you actually request the data via AT+USORD it buffers data it received. Similar with sending data, you tell the modem via the 2nd parameter to AT+USOWR how much data you are going to send. Depending on the internal implementation, it may buffer your data locally until it receives the full payload and then begin sending data packets over TCP. With PPP it may behave differently as it does not know the full payload size. Basically I could see the sum of all these delays for send and receive being like 400-1000ms on top of network latency. Not bad for most applications but if you needed to get as close to realtime as possible this may be unacceptable.

Your experience with buffering
Check out the very bottom of page 195, top of 196 in the at command manual: https://www.u-blox.com/sites/default/files/SARA-R4-SARA-N4_ATCommands_(UBX-17003787).pdf

If no network signal is available, the TCP packets are enqueued until the network will become available again. If the TCP queue is full the +USOWR command will return an error result code. To get the last socket error use the +USOCTL=1 command. If the error code returned is 11, it means that the queue is full.

I believe this is the behavior you saw. The modem keeps the socket open until explicitly closed by you or explicitly closed by the server, or likely some long timeout (few minutes+ I would guess). This means when you pull the antenna, the socket remains in the connected state so the module will buffer the data locally. EDIT: Note you can check the status of your socket via the AT+USOCTL=#,10 command where # is the socket number

I dont see explicit answers in the AT command manual for your following questions but my guesses and rationale are below:

under what circumstances does the module buffer? When a TCP connection is open but no network connectivity is available (per AT command manual).

How large is the buffer? I would guess enough for a few kB of data, like 8kB+. You can test this by trying to send larger and larger packets with the antenna removed but TCP established.

Under what conditions will it discard the buffer? I would guess only when the connection is closed either via AT command, explicit close connection command from the server, or the many minute+ timeout I assume exists.

Is it possible to clear the buffer Best way would be to close the socket via AT command and you can immediately re-open it to try again without a full buffer.

1 Like
15

Hi Reuben, thanks very much for following up. Let me share my thinking given the information you shared…

Under Linux, it is usual to launch the PPP daemon (/usr/sbin/pppd) to create a ppp connection. In the configuration file for pppd (/etc/ppp/options) a “chat script” file is typically provided. pppd uses the chat program to “chat” with the modem (send it commands and handle responses). The chat script contains the AT commands (including ATD…) which should be sent to the modem.

So, the problem is that pppd opens the dev attached to the modem (e.g. /dev/ttyUSB2). Since pppd opens the dev, it cannot simultaneously be opened by another program. So, it seems like there are two possible ways to send the escape sequence (+++) to the NovaM:

  1. modify pppd, writing in a hook, to enable it to send +++ to perform the switch, another hook to allow interaction with the modem after the +++, and a third hook to switch back to Data Mode. While this is theoretically possible, it is a daunting task and it’s probably not a great idea to create a variant of a basic utility.

  2. don’t use pppd. Instead use AT commands to do everything (as @AndrewGifft described). The dev must be opened to send the AT commands so, of course, +++ can be easily sent to the opened dev.

Does it seem to you that I’ve got this right?

You know, I started off using the Hologram SDK and CLI. I backed off that to set up my own ppp connection. Now I’m backing off that to simply control the NovaM with AT commands. Pretty soon I’ll back off the Nova and my tracker altogether and just have the bus driver yell his location out the window. :wink:

Cheers,
Matt

1 Like
16

Am I correct to assume that once you establish the ppp connection you have no way to switch from Data Mode to Online Command Mode?

17

Communicating with U-Blox is certainly a good idea, but they won’t give me the time of day.

18

Okay, after sleeping on the idea of handling all data communication myself using AT commands, I realized the biggest drawback: my firmware is multi-threaded with one thread sending location data, another periodically checking for software updates, another requesting directions when needed, another periodically sending driver hours, etc., etc.

There is no problem letting each thread do it’s job when there is a ppp connection. Linux uses the connection and takes care of all the details. Avoiding ppp and writing a solution to handle each threads needs is a complex design matter which will likely include bugs which will be devilishly difficult to sort out. Also, as the devices are deployed all over the place, updating for bug fixes will not be much fun.

Also, as I think about it, the main challenge is not to able to send AT commands whenever I want (thought that is good too), but rather to determine–as soon as possible–that the cell connection has dropped so the firmware can decide what to do.

The main wrinkle has been that I’ve been sending location data via UDP. My thinking has been that since my data payload in a location packet is only 17 bytes, the additional overhead of the TCP packet header (bigger than a UDP header) and the TCP handshaking will increase my data consumption by more than 100%. Also, my experience, after sending millions of UDP packets is that a small fraction of 1% of packets get lost (I cache packets while there is no connection and send them when the connection is regained).

What I’m finding with the Nova is that it is very difficult, once a ppp connection is established (and I can no longer send AT commands), to determine when the ppp connection drops. Running ifconfig will show ppp0 long after the connection is gone (5+ minutes). Checking /proc/net/dev is similar (I think ifconfig gets its info from /proc/net/dev).

The discussion of buffering seems to indicate that the R410 module even keeps it a “secret” that it has lost the connection and buffers data hoping the connection will return. Given the desire of the module to “hide” the dropped connection, I think no error occurs when a UDP packet can’t be sent to the carrier and, of course, once a carrier sends a UPD packet there is never any kind of acknowledgement of receipt.

All of this suggests that using this module I’m not going to be able to send UDP packets and know, in a timely fashion, that the connection has dropped.

So, my latest thought is to use TCP to send location information. If I don’t get a prompt acknowledgement (within, say, 10 secs) then I can interpret this to mean the network connection was lost and the last packet sent should be cached for later sending.

Do you have experience with this? When you send a TCP packet do you get accurate acknowledgement? That is, if there is no error, does that mean it was delivered to the network and a response returned by the destination server, or is the module giving a false ACK while it buffers the data for later sending?

If the module is sending false ACKs then of course, I can have my server send back a byte of data indicating success and then I’ll know for sure, but if I don’t need for the server to send back info then why waste the data…

Obviously, I need to perform my own tests, but if you have any experience to share I’ll be most grateful.

-Matt

19

Yea at the end of the day, you have to change something about your implementation. It can be either further debugging ppp, trying to work with U-Blox to get online command mode and switching states to work, it can be writing a thread-safe wrapper around your COM port to send / receive data via AT commands for multiple threads, or it can be single threading your application and doing event based or simple periodically checking of states. Unfortunately “none of the above” doesn’t seem like an option.

Depending on your comfort level with writing thread-safe applications writing a wrapper over around the COM port would allow you to keep most of your software in place.

Or since all of your events are relatively slow for computers (as they are driven by human input and vehicle location) you could just have a master thread or everything in one thread and just check states of everything at like 20Hz. No one will notice if a command to turn is 50ms later, or if you have 50ms jitter on your location positioning.

Lastly, although it is somewhat annoying, I see why the U-Blox device implements buffering. Cellular networks, and all wireless networks are somewhat unreliable, in most cases, if you loose signal for 2s and then get it back, you would want everything to just resume gracefully as if nothing happened. I would rather a web page just takes an extra second to load than to error out, similarly I would rather a phone call just have 1s of silence and resume then to drop the call just because the network went out briefly. Now maybe there should be a configurable timeout, like “if I am trying to get online for 30s and still nothing, then give me an error”.

Answers to some of your questions

Question All of this suggests that using this module I’m not going to be able to send UDP packets and know, in a timely fashion, that the connection has dropped.

Answer You can do this through AT commands to atleast check network registration state just prior to sending the UDP packet. Especially useful if you use AT commands for everything (including sending data)

Question When you send a TCP packet do you get accurate acknowledgement? That is, if there is no error, does that mean it was delivered to the network and a response returned by the destination server, or is the module giving a false ACK while it buffers the data for later sending?

Answer See AT command manual middle of page 196 for AT+USOWR:

The information text response indicates that data has been sent to lower level of protocol stack. This is not an indication of an acknowledgement received by the remote server the socket is connected to.

Basically when you get OK response to AT+USOWR... command it means the modem accepted your data, not that it has been successfully sent and received by the server. I don’t think the socket will give a false ACK and you can check the status of the TCP connection with the AT+USOCTL command.

Question If the module is sending false ACKs then of course, I can have my server send back a byte of data indicating success and then I’ll know for sure, but if I don’t need for the server to send back info then why waste the data

Answer This may not be required if the result of AT+USOCTL is sufficient for you, that being said, I would have your server respond, it is an easier way to know at the application layer that everything went through correctly.

Additional Thoughts

  1. TCP does have a decent amount of overhead especially for small payload sizes, what confuses me though is that you must have a lot more than this going on right? You said you have multiple threads running in parallel checking for updates, sending this lat/long/speed/etc data, gathering turn-by-turn directions (I’m guessing you do this via some google API), speech to text (Are you doing this all on the device or using some cloud service like Alexa). If so its seems this overhead on only one of the types of communication going over this link is unnecessary optimization. Also your data consumption must be relatively high anyway if all this is going on.
  2. If you control the server for these messages, which it sounds like you do, and you are sending data quickly you may be able to keep the TCP connection open on both sides, atleast reducing the relatively high data cost of establishing and gracefully closing a connection.
  3. I’m not sure why it is important to buffer data and detect if the network is out. For your use case what seems to matter is “where is the bus now” not “where was the bus before”. So the only bit of data that is important is the most recent right? If you don’t have network connectivity the data collected during that time is not valuable as it will be stale by the time the network comes back up?
  4. Lastly, although its long and boring, I would suggest reading all of section 7, 13, 17, and 18 of the AT command manual. It really helps to understand what you can and cannot do with the modem related to network communication.
20

Again, thank you very much for your thoughtful and informative reply.

Let me respond first to some of your “Additional Thoughts” as they will provide some project context.

Additional Thought 1

My tracker sends location info every 30 seconds while the bus is running (it can detect whether or not the bus running). I send the locations via UDP to minimize data consumption.

Turn-by-turn directions are done entirely on the device. Text-to-speech is done entirely on the device. Once a day (or so) the device checks in with the server to see if an update is available. Usually there is not, so that’s one small packet per day. Once in a while the tracker checks with the server to see if the route the bus runs has changed. Again, usually not, so one packet. I use UDP for the location packets and TCP for everything else.

On a typical school bus, my tracker consumes about 1mb of data/month. Switching from UDP to TCP will, I expect, increase my data consumption by 100% or so.

Additional Thought 2

I send locations every 30 seconds or so. With UDP, I wrote a simple server which listens for incomming packets, plucks the data from the UDP packets and waits for the next one. With TCP, keeping the connection open shouldn’t be difficult, but there can be many thousands of buses so I have to think through the consequence consequences holding all the connections open for 6+ hours per day.

Additional Thought 3

Yes, it would seem that only the current location matters, but that’s actually the case. For a parent or student tracking her bus, only the current location matters. But my customer (the one who pays) is actually the school district’s transportation department. A key feature for them is that the system stores all location data for the entire school year. The system produces graphical traces (markers on a map) of the historical path of any bus during any period. It can produce a wide variety of reports (miles/hours operated for each bus, excessive idling, speeding, etc.) all by analyzing the historical location information. It can even determine where the actual bus stops are located and the details of the bus route, again based on the historical location info. So, losing on occasional location packet is no big deal, but losing more than that can interfere with the quality of the reporting and analysis.

Multiple AT Command Interfaces

In this document:

https://www.u-blox.com/sites/default/files/SARA-R4-SARA-N4_ATCommands_%28UBX-17003787%29.pdf

Section “B.5 Multiple AT command interfaces” (p. 309) indicates that more than one program can access the device’s AT commands. It is a bit terse and, of course, there is no example. Have you come across this? Do you know anything about it?

An Entirely Different Solution

There is another possible solution, which you raised early on: put a U-Blox (or other) module directly on my custom circuit board. On p. 11 of the same document, it says:

SARA-R4 / SARA-N4
u-blox cellular modules can implement more than one interface between the DTE and the DCE, either
virtual interfaces (multiplexer channels) or physical interfaces (UART, USB, SPI, etc., when available).

The Nova device does not expose additional physical interfaces, but in a custom design one could. Do you know if what they are saying is that, for example, using UART and USB interfaces it is possible to access the module from two separate programs?

Again, thanks for your thoughts and suggestions.

21

Ok I think I understand your system better. It sounds like 99.5% of the time only one “thread” or application needs to be on the network, this is your 30s interval of location information. Then 1x per day you might need to download updates via one or a few files for your updates. How long would this take, 10-60s? Also “once in a while” (like 2-5x per day?) you need to check for route updates. I am betting this would take less time to download, like 5-30s. It seems like synchronization between these tasks would be pretty easy. Also it seems network communication isnt really critical (in the sense the bus driver wont miss a stop, or miss a turn if you cant send/receive data).

Why not have one thread that does all your necessary communication? One dumb way to do this could be to have your gps thread just dump location data every 10-15s to a file like YYYY_MM_DD_HH_mm_ss_latlng.json, then every 30s your data communication thread looks if any files exist and sends their contents to your server. Similarly if a file “DOWNLOAD_UPDATE.txt” exists your code will check and download an update and store the new files in a certain location. Lastly if “CHECK_ROUTE.txt” exists, it will try to download a new route and store the new route file somewhere pre-defined. Now there are smarter ways to signal between threads / processes, local sockets being an easy one but you get the idea. If my timing estimates are correct, at worst you will have 60s between location messages to your server vs 30s, you could also build in some smarts such as you only “CHECK_ROUTE” or “DOWNLOAD_UPDATE” if the bus isnt on or if no latlng data is present.

Honestly if I was building this product, that’s how I would implement it, you don’t really need parallel network access.

As for your thought about using multiple interfaces to the device. This could work, I have never done it, and there are warnings about sending conflicting settings on multiple command ports at the same time. So you will likely need one “master” controller that does things like initialize and setup the modem and ensure your other ports are only used for data communication.

22

Hi Andrew, Thanks for your further thoughts on this matter.

Since I’ve written the whole system in Java, and I’m using multiple threads (not processes) the threads can communicate directly with each other, so there’s no need for creating files to represent various requests. I can easily write a class called “Communicator” which accepts requests for various communication services, queues them up, and lets the corresponding thread pause (if it requires a response) until the result it desires is obtained by Communicator and returned to it. Requests to Communicator can also specify whether to send by TCP or UDP. This is certainly doable.

Regarding multiple interfaces to the device, I think if it is possible it is better approach because it doesn’t require interfering in each thread which requires Internet access by recoding it to use the Communicator class previously described. It seems a shame to make significant changes to an otherwise clean solution just because of the limitations of the Nova. Presently, my tracker already works with some other USB modems (all 3g) and it is certainly imaginable that I will use other LTE devices which are not the Nova. I hate to beat up a system just to accommodate a piece of hardware which isn’t quite up to snuff. I’ll mention that I have deployed many ZTE mf190 GSM USB dongles. They cost about $8 and I can send them AT commands while a ppp session is active.

Also, I don’t think a “master” controller would be needed for the multiple interface solution. Rather, I think it’s a straightforward synchronization problem.

Anyway, I appreciate your thoughts. It’s very helpful to kick around ideas…

23

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.