Continuous, Indefinite Connection


#1

Hi Hologram Community,

I’m creating a hologram application where I want to be connected to the hologram network effectively indefinitely.

I have run into lots of issues with losing the hologram connection and crashing when I try to reconnect. I’ve worked around lots of these issues, but I still find that the server I’ve created is unreliable and can’t last more than a few day in the field. Can anyone comment on success they’ve had with different methods of maintaining a Hologram connection? Methods I’m currently aware of include:

  1. Calling disconnect and reconnecting
  2. Killing the old connection and creating a new one (what I do now - more low level issues than I expected)
  3. Killing the whole process and restarting with a cron job).

I know this question has been brought up a lot of times on this forum, but there doesn’t seem to be a good community answer or a response from the Hologram team. I would really love the if the Hologram team could provide best practices for this situation.

Thanks in advance!
Alex


#2

Hi @Alexander_Kaplan - Thanks for the feedback on your experience keeping a network connection up. A few questions to help dig a bit further:

  1. What device are you using to connect and how are you starting up the connection?
  2. Once connection is up, how often are you transmitting data over the connection over that time period?

Thanks, hope we can work together on a solution.

-Ryan


#3

Hey @ryanHologram thanks for getting back to me so quickly!

I’m using the Hologram Nova with a raspberry pi and I transmit data every ~5 minutes.


#4

Great some more questions:

  1. Do you have any logs of what happens when you try sending and the connection drops?
  2. Is this the CAT-M1/NB R410 or 3G U201 Nova?
  3. Also, how are you setting up the initial connection and how are you sending the data? e.g. AT commands? python script with SDK? CLI on cron job? other?

-Ryan


#5
  1. Unfortunately I don’t have comprehensive logs, because I’m not sure what information information is available to me. However, generally what happens is a message fails to send with an error code == ERR_UNKNOWN, then the Nova disconnects. When it disconnects, I have to determine and kill the process associated with the hologram cloud connection before reconnecting or I get a too many connections issue (this will crash the server). Occasionally this reconnection process doesn’t work, which is why I’m looking for best practices. I would love to maintain a more reliable connection or to be able to reconnect using the same hologram cloud instance.

  2. 3G U201 Nova

  3. I have written a python script using the SDK which sets up the connection, then runs indefinitely sending data using the sendMessage method. I’m considering incorporating cron to restart this sever periodically.


#6

Hi Alex,
I have been having the same problem with my server, so I changed SIM’s to the actually network Hologram is using. It resulted with my server running cron job without any interruptions. The packages don’t seem to transfer the same using Hologram, I have not been able to connect Hologram network for 22 days now with my module.

Jeremy


#7

@Contemporary_Mgmt I dont think you can use hologram cloud with a device that does not have a hologram simcard, so if you switched SIMs away from Hologram you will have to use a different cloud messaging service (AWS and Azure have some).

I’m also confused what you mean with “my server running cron job without any interruptions” but “I have not been able to connect to Hologram network for 22 days…” You first sentence says everything is working but the second one says you have no connection to Hologram?


Hologram SIM Timeout for UDP (used on Particle hardware)
#8

Hi Andrew,
I noticed that afterwards, but left it because it sounded network related still. Are you saying that the SIM and Hologram Nova aren’t using the same towers? I would think that probably are, right? I was able to run the job on Hologram a couple of times, but lost packages or lost signal. My point is only that, I ping the tower then bought a new SIM from the company that supposedly owns the tower Hologram was using and never had an issue again. Why is there a difference? Seems Hologram is not being treated as a priority.


#10

Hi @Contemporary_Mgmt - Would be interested to hear more about how you’re using the Nova and differences you’re noticing across SIMs. You’re right that our network will attach our partner providers’ towers to enable global coverage however the difference in quality you’re describing is not standard, especially not connecting for 22 days. Can you e-mail success@hologram.io with additional info on your specific SIM #, module, and how you’re connecting with the Nova to your server? I’d like to escalate it with our success team to investigate further.


#11

Hello Everyone,

Thank you for the feedback and all the helpful comments on this thread. I wanted to chime in and provide some more information here to help everyone understand the issue and how to resolve it. At its core, the cellular network was designed for mobile communications with devices that moved around and switched from cell to cell, it was not designed for long term static connections. For this reason, and to preserve their resources, cellular towers will automatically tear down sessions that have been inactive for some time and have not been properly terminated by the cellular device. This behavior is universal across cellular towers and carriers and is not something unique to Hologram SIMs. Now, this becomes an issue when you want to have continuous inbound access to your device like you would if it were using a wired connection or WiFi.

There are a few different approaches to keeping your device connected. One would be to have your device send keep alive messages to avoid the tower deeming your device as inactive and tearing down it’s session. Otherwise it’s common for cellular devices to have to implement a watchdog to reconnect when a connection is lost (your cellphone does this all the time, for instance). The use case and device characteristics will likely determine which method is optimal for your specific situation.

For users using our SDK, please note that the send command should be use as a stand alone command and does not need the device to be in a PPP session by issuing the connect command.

For anyone using Cat-M1 I really like @AndrewGifft’s answer in this thread.

Best,
Maiky


Project interfacing with AWS
#12

Thanks for the clarification @Maiky! Any idea what the necessary time frame is to prevent the session from being lost?


#13

Hi everyone. I have many devices in the field and for all extensive purposes have durable TCP connections. The towers do dump the connections often or seldom but unpredictable, even with persistent pings to keep things alive. The reason is the towers, when over loaded drop connections.

Nova, using Linux on a Pi.

What I have done to deal with the lost connection issue has three(3) major components. First to keep the host software running I use a custom created server that is thread based. When a connection happens the thread with the connection is detached and left on its own to service the client. If anything happens to the connection eventually eventually the connection times out, the thread reads the error and exits, linux will discard the thread and system resources are returned. If the connection completes without issue the same is true. Linux is good at handling terminated threads. Any shared resources need to be properly mutexed.

Second, the cell connection is established using the sudo hologram modem connect. This creates a ppp session where the system will listen to incoming connections from the hologram network. That service allows my server to receive incoming connections from the cell network. At the same time using the Hologram Dash Board each of my units has tunneling enabled and clients connect through ssh port forwarding. All in the hologram docs.

Third, I have a cron job that runs once every minute that’s sole function is to examine the network connections looking for the hologram ppp network service. If it is not found, it logs the lost connection and then attempts to reconnect by issuing the sudo hologram modem connect command. The job will try two attempts and then exit. Only two attempts because the job runs every minute, the process will be repeated and eventually the network will connect.

This solved my issue until hologram had an outage that affected the clients connecting through the tunneling services.

My systems have run for two years without any issues other than just described. In some areas of the US the lost connections are seldom, in other busy areas the lost connections are more frequent. However looking at the remote access to the systems, other than when hologram was down, the connections appear to be durable. In other words, lost connections are quickly recovered to the point that lost connections are only noticed when looking at the logs.

I would not put life critical systems on these connections, nor would I do that with any cell connection. But the lost and recovered connections look like ordinary connections with network traffic delays from time to time.

I like the connections I have. I am even able to view and download large logs from my devices, serve up html status pages and control lots of operating parameters. Just as if I had a hardware network connection.

The Hologram network does work well!


#14

Hi Maiky,
Are you saying we can’t use the Hologram network with our own API? I tested what your saying by pausing the data with the partner, and then activating it while restarting module. Good news it connected and ran; however, I made another run within minutes of the successful one, but ended in an error again. I agree with you about the partners terminating the connection, but unable to agree it’s due to inactive device when I’m running a cron job every minute, or running it manually every other minute, I don’t seem to need a “stand alone command” using another partners SIM card. I’m sorry, but I’m not taking sides here or it would be Hologram’s because I want it to work.