Please excuse my lack of knowledge:
How would you send out a ping from the Nova? With AT commands?
Would you ping at the same time that the Nova is in receive mode, or directly before going into the receive state?
Yep, we totally agree which is why we want to really get this testing.
Definitely understand your frustration here and we do want to get this working for everyone.
You’re correct about the different power down states on 3G making this a non-issue there.
As far as NB, we’re still in early stages with supporting NB with our SIMs and so this feature doesn’t work at all on NB yet. It’s unclear if it can be supported since NB is incredibly stripped down. We won’t know that until we’re farther along with our NB discussions with carriers.
Oh you can send via AT, but I just meant doing a hologram network connect and then something like ping -I ppp0 8.8.8.8
I guess you’d have to do the receive manually at that point though since the modem will be operating over PPP so you could try receiving on port 4010 with the linux nc utility.
I just simulated by opening up a couple terminals on my Pi, then running hologram network connect and then nc -l 4010
I sent a message from the dashboard and it didn’t show up. Then I ran ping -I ppp0 8.8.8.8 in another terminal and the messages started showing up immediately
@Reuben are you talking about eDRX / PSM (see: LTE eDRX and PSM Technology Explained for LTE-M1 | Link Labs) when talking about:
I think on most modems both are disabled by default, atleast on the Sara R410M and monitoring the current draw of my R410M based boards I can see the current spikes every ~1.28s for the paging cycle.
If it is one of these features a fix should be to explicitly disable them right?
Yes, I think that would be it. I believe the network itself has an ability to assign timing there so I wonder if maybe some operators are using different values which might also explain some of the variation we’ve seen. I’m curious with your quick paging cycle if you see the same phenomena that I’ve described in here.
That page is also very interesting to me since it still claims a paging cycle in seconds even with M1 which shouldn’t cause a problem like this. This seems to conflict with some info we’ve gotten from some carrier partners. We’ll have to do a little more research there. I wonder if there’s a difference here between when it can receive TCP/IP vs when it can receive SMS or something more at the network level.
Blank response basically:
AT+CEDRXS?
+CEDRXS:
OK
ATI command for reference:
Manufacturer: u-blox
Model: SARA-R410M-02B
Revision: L0.0.00.00.05.06 [Feb 03 2018 13:00:41]
SVN: 00
IMEI: 3527xxxxxxxxxxx
OK
Response from AT+CEDRXRDP:
CEDRXRDP: 0
OK
Here is a plot of my device current, note there is an LED that blinks with a period of 1/2s which is the lower amplitude square wave, you can see the network pings at ~1.25s. This behavior persists indefinitely on these devices (have had them up monitoring current for days, same behavior).
1 Like
Interesting. Yeah it is disabled on yours but on mine it is not. I never did that intentionally so I think some networks might be pushing that setting.
Though I can disable that if I want to.
We just did some more testing around here. If I do turn it off, then sending messages no longer has an issue.
With it on, we’re seeing that the TCP SYN packet needs to be retried a bunch of times and with the TCP backoff that means it takes longer and longer to retry so even though my Nova is only going 9 seconds between tower connects, it ends up taking a lot longer for the message to hit it since we’re basically waiting for the timing to line up right.
So anyway, this does show that the new features built-in to Cat-M are the cause of this but we think it also points to an issue with some of our carrier partner’s networking settings and we’re going to be reaching out to them.
In the meantime, the fix we were already working on should make this more reliable and it sounds like we have a new workaround which is to turn off eDRX on the board to speed things up as well. We might build that into the next version of the SDK receive command.
Great to hear, I will be adding that command to my initialization as well, although it would be interesting to know if the carrier can override that setting and re-enable (meaning it would have to be re-disabled periodically).
Hopefully AT+CEDRXS=3 actually disables eDRX and does not allow the network to re-enable. Note this setting is not saved in NVM so needs to be re-issued each power cycle.
Hey @Blake we deployed the update this morning. On top of increasing the timeout, it will give better error responses now about what actually went wrong.
That being said, I am still seeing some messages get missed when eDRX is enabled so we may need to tweak things further. We’ll keep looking at it. I think at this point, we’d recommend disabling eDRX on the modem if you want to be able to receive stuff without any issues.
You can do this by issuing the command AT+CEDRXS=3
We’ll probably build an easier way to do this into the SDK at some point in the future.
Note that this will increase overall power usage a little bit.
Oh, I should also mention that the R410 can be a little slow just to close the socket at the end of the receive for some reason so it can take a minute for the message to print out.
@Reuben @AndrewGifft Thanks for the information and updates. I’ve been away from the project the past two days but will try out the updates this weekend.
Does the update affect whether you need to disable eDRX each power cycle, or will it need to be re-issued each time as Andrew indicated?
I’m very new to cellular communication. I’ve googled around on this question but can’t find a straightforward answer: When using eDRX, the radio is off, disconnecting the device from the cell network. It seems like this can be anywhere from 10 seconds to up to 40 minutes or longer between check-ins. If data is sent to the device while it is disconnected, what happens to that data? Will the network wait for the device to come back online and then send the data? Is it lost? Is it up to the sender to keep track of the device’s ability to receive (like pinging it first or something)?
Thanks.
Edit: I see this:
It is recommended that a “store and forward” policy should be supported for PSM. The
operator should consider storing/forwarding the last received packets or an SMS (whichever
is supported) to be sent to the device when it awakens. At a minimum, the last packet of
100 bytes should be sent, to allow the customer to send a simple message. Any
store/forward limitations should be communicated to the customer as part of a service level
agreement.
In this document:
https://www.gsma.com/newsroom/wp-content/uploads/CLP.28v1.0.pdf
Would the operator in this instance be AT&T, Hologram, or my sending device in this instance?
Yeah, so if data is sent while disconnected at the moment, it really only gets stored for maybe a minute or so until the usual TCP timeout. In this case, the store and forward policy is kind of a combo of different operators so that’s why it’s a little messy right now.
The update we pushed does not affect whether the local operator can try to suggest that that modem use eDRX so you might still need to send that command occasionally. It’s a little unclear on when that switch might happen by itself. When I turn it off on my modem now it doesn’t seem to always get turned back on. It might only be when it sees an operator for the first time or something.
Sorry can’t give super definite answers to these questions. As you may have seen from other threads, Cat-M1 is still pretty new and there’s still a lot of policies that are getting formulated as more carriers add support.
Right, but wasn’t today’s update meant to increase the TCP timeout?
Did it get increased to a minute from something less? Or what is it? Because, if I understand correctly, the DRX sleep cycle (or minimum eDRX cycle with 1 hyperframe) is 10.24 seconds, which is less than a minute. So shouldn’t it theoretically wake up and get the message before TCP times out?
Yeah so it seems like there might be different timeouts at different levels of the system. From what I’m seeing, some carriers are not actually storing packets for the entire TCP timeout and might only be keeping them for a couple seconds. The reason increasing the TCP timeout on our side (which we did to 90 seconds) helps is that on top of getting the carriers to try to store for longer, it also allows us to keep retrying during that timeout window so even if they’re not doing store and forward, we can hit the time window.
Basically, we’re trying to workaround some weird configs with the local operators.
We’re also going to continue to work with the operators and tweak things on our end so things should keep improving.
Got it, thanks. I’ll try it out and let you know if I see a difference with receiving messages after the update.
So I just tried to send TCP data to the device (pi3b+/nova) and it appears to be working now. So that’s a good sign.
I do have a question though:
I have been using the command line to operate the modem, which means I put the Nova into receive mode before trying to send a message. Wouldn’t putting the Nova into receive mode automatically bring the radio out of any sleep mode/eDRX it may be in? The Nova is only able to receive messages while in receive mode correct?
Is receive mode meant to be the default state that the radio spends most of its time in, except when it is sending messages?
Thanks.
Just wanted to follow-up on Reuben’s comment with some more information after several of us discussed this thread.
Support for Cat-M1 features like eDRX and other power saving modes on inbound connections is a little bit of a multi-faceted problem at the moment, with a combination of simultaneous issues :
- Cat-M1 deployments are relatively new to several carriers, and still being rolled out for others
- network equipment vendors and gateways are catching up in deploying extended power saving support at the packet delivery level
- many networks do not currently implement the store-and-forward features proposed for M1 deployments
- specifications around latency and other timings diverge from those of 3G and higher-category LTE
We are finishing the full implementation of eDRX and necessary network changes to accommodate modified timings needed for modules to wake up from Cat-M1 power saving states and receive inbound socket connections. These enhancements for Cat-M1 provide a similar experience as store-and-forward support would have provided, allowing inbound connection requests to go through quickly and successfully even for modules that were in an idle state on the radio channel at the time. This does not affect device-originated outbound connections, or the ability to leverage power savings in general, which are implemented. The upcoming improvements specifically address the waking of modules with device-terminated inbound connections (e.g. SpaceBridge or packets sent via API) and the ability to ensure the opening of a socket connection succeeds quickly and successfully with better handling of edge cases.
We pushed a preview of the upcoming improvements this week to help improve the experience for those who have programmed modules to listen on a socket over Cat-M1 and who were experiencing connection timeouts, by adjusting timings and improving error messages to be more helpful while troubleshooting. This is one of several improvements that we’re pushing out for Cat-M1 inbound connections support as we accommodate trending and now-maturing Cat-M1 operator deployments, noting that we are pushing changes out in phases, like the first one this week, to afford ample testing across a variety of operators and regions. (Also note that these changes are, again, only related to connections originating over SpaceBridge or our API and terminate on sockets that devices are listening on, but do not affect device-initiate outbound connections.)
1 Like
Receive mode is more an abstraction by our python SDK where it uses AT commands to open a socket on the modem and listen on it. It is not the normal default state of the modem and the ublox does not automatically disable eDRX when listening on a socket
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.