Data Usage and more


#1

I have a simple application and I just want to confirm what I did.

Python Code
while 1:

hologram = HologramCloud(dict(), network=‘cellular’)

result = hologram.network.connect()
if result == False:
print(’ Failed to connect to cell network’)
else:
print(‘Cellular Attached’)

mydata = {‘customerid’ : 100, ‘fuel’ : 71, ‘temperature’ : temperature, ‘humidity’ : humidity }
dataJSON = json.dumps(mydata)

response_code = hologram.sendMessage(dataJSON)
print(response_code)
sleep(60)
hologram.network.disconnect()
print(‘Cellular Disconnected’)

Data Received---------------------------------------------------------------------
{“data”: {“fuel”: 71, “temperature”: “74”, “humidity”: “60”}, “customerid”: 100}

My Routing--------------------------------------------------------------------------

{customerid : <<decdata.customerid>>,data : {fuel : <<decdata.fuel>>, temperature : <<decdata.temperature>>, humidity : <<decdata.humidity>>}}

Also, I quickly reached 1 Meg and my SIM was paused, Is this sent via HTTPS, I cannot believe how quickly I reached 1 MB

Thanks


#2

Hey Phil_Ferro,

I’d recommend using tools like, wireshark, bmon, and ifconfig to check data usage. Hologram’s messaging protocol is designed to be as light weight as possible so it seems like the data you used came from something else.

The situation you describe is one that we have found commonly caused by device heartbeat data and background data. The Nova is an Internet gateway so any other internet activity on the device, even outside the CLI and Python SDK might have been transmitted through the Nova.

As an FYI, you don’t need to use the connect function to send messages using sendMessage, removing that should prevent other programs from using your data.

Best,
Maiky


#3

I have a similar experience, I am using the sendMessage function only to send a simple 2byte piece of data. I’m sending this 96 times per day, however I’m seeing data charges of about 100kB per day, I don’t know what 99% of that data is going to but I should only be seeing 1kB or so per day considering a little overhead on the 200B of actual data. Where can we look to determine how big a data package is going to be when using the “hologram send” function? I don’t have any other background data use on this Linux system.


#4

Note that setting up a TCP connection requires “significant” data and sending a http request/response uses at least 500-700 bytes (in my experience). If it is using https expect 1-3kB per connection. With many uses cases the tcp/http connection and teardown is by far the dominant driver of data usage not the payload. You also have the dns lookup of the domain name which uses 100+ or so bytes.

Note I ran the dns lookup for the hologram socket and got the response (245 chars):

;; Truncated, retrying in TCP mode.
cloudsocket.hologram.io. 300    IN      CNAME   csr-entry.prod.teamhologram.ninja.
csr-entry.prod.teamhologram.ninja. 60 IN A      34.206.148.15
csr-entry.prod.teamhologram.ninja. 60 IN A      18.206.138.90

Briefly looking at the Python SDK it looks like hologram uses a new tcp connection (to a domain name which requires dns lookup) for each message. You can look at the details of the messaging I think they use here: https://hologram.io/docs/reference/cloud/embedded/

In summary, I am not very surprised at 1kB per message. The two ways to reduce data usage is:

  • Buffer data and send fewer messages (for example send 10 messages a day with 10 readings in each message)
  • use a protocol with a persistent TCP connection like MQTT (I dont know if hologram cloud supports this)

For my stuff I go with the first bullet and just buffer data locally and send only a few messages.
EDIT: Note I also have my own servers with static IP address that I connect to using IP to potentially save the DNS lookup although this is does not use hologram’s backend so may not be a viable solution for you.


#5

Thanks for the suggestions Andrew! Definitely two good approaches to reduce data usage. At the moment we don’t have an MQTT solution for the hologram cloud, but we have seen users set up MQTT using our network, but not our protocol. In fact @benstr has a series of youtube videos on this.

As an aside, here is an article where one of our engineers compares our messaging protocol vs sending messages via an HTTP call. I think it does a great job illustrating how much overhead is reduced, but like Andrew explained there are ways to further reduce that if your use case can support it.


#6

That article is great and should be useful to @electriclove especially (note the cloud/socket billed size vs payload size):

image

Where I would clarify and say the article isn’t quite fair is calling “HTTPS” “HTTP” HTTPS requires SSL, cert handshakes and encryption and increasingly larger handshakes and more complex (ie larger) encryption as the standards mature. Raw HTTP/1.1 is basically TCP (same as coud/socket) with some standardized text headers and is much more comparable.

HTTP and the Cloud/Socket offer no encryption meaning anyone in the middle can see, spoof, or alter the data but this lack of encryption makes it much more compact and easier to implement. HTTPS which is what Hologram’s and many other’s REST API’s use does offer encryption at the cost of ~5kB / request. So the conclusion is more “due to how the internet works, if you want a reliable (ie TCP) data transfer expect ~500 bytes in overhead, if you want a reliable and secure way to transmit data expect ~5kB overhead”


#7

OK Thanks, excellent responses, I will remove my “result = hologram.network.connect()” command and trim my data, I will try again next month.

Thanks!