A message queue for the abhorrently named "Internet of Things"

As you may recall, I have hooked up various parts of my house to a computer so I can control them remotely. To do that, I have an HTTP server for which I open a port, and everything runs on this server, but what if I need to add another device? What if I need to have multiple devices listening for a command?

Having to open a port for each and every one of them, exposing them to the internet and configuring all this is a huge hassle, a security problem and very brittle. Wouldn’t it be much better if there were a centralized message queue where I could post messages and have an arbitrary number of devices read them?

It turns out, the excellent folks at dweet.io have done exactly this. They have created a fantastic service that you can just write to, no API keys or any setup required. You subscribe to a key either by polling or just by listening to the URL, and publish by POSTing to a key in another URL, and that’s it!

However, I have two problems with the service:

  • I don’t control it, and since whoever controls it will be able to send arbitrary messages to devices in my house, this is a rather major issue.
  • I can’t run it on my server, so latency will be a problem. I can use the live endpoint to minimize polling, but it requires a bit of extra complexity on the client, which I’m not sure about.

Looking around, I also discovered data.sparkfun.com, which is another great (and free) service, and is also open source, but it’s node.js, and I didn’t feel like dealing with that whole hassle just to deploy one small service.

So, I did what any self-respecting developer would do, and wrote my own. In Go. Thus, Gweet was born.

Gweet

Gweet is a really small service written in Go, which duplicates the functionality with dweet.io, but with a less insane API. Before I continue, you can get it from its Github page:

https://github.com/skorokithakis/gweet

The API is very simple. It doesn’t have live streaming for now, but the latter is so useful that I’m thinking of adding it, maybe even today. I added it. Read on for details. Here’s how you can publish and poll for data:

POST http://gweetapp.appspot.com/stream/test/?foo=bar

{
  "message":{
    "created":"2014-11-29T13:38:41.392412429Z",
    "name":"test",
    "values":{
      "foo":[
        "bar"
      ]
    }
  },
  "status":"success"
}

GET http://gweetapp.appspot.com/stream/test/

{
  "messages":[
    {
      "created":"2014-11-29T13:38:36.496314194Z",
      "name":"test",
      "values":{
        "foo":[
          "bar"
        ]
      }
    }
  ]
}

Simple as that! Queues show the last 100 items by default, and get deleted if nothing gets published for a day. Feel free to change these defaults to suit you, though.

Live streaming is implemented in dweet as HTTP chunked, which is a pretty simple format and makes a lot of sense, so I may give it a go. I have now implemented HTTP live streaming, and it’s awesome. Here’s how you use it:

GET http://gweetapp.appspot.com/stream/test/?streaming=1

{"created":"2014-11-29T13:38:36.496314194Z", "name":"test", "values":{"foo":["bar"]}
{"created":"2014-11-29T13:38:38.289783273Z", "name":"test", "values":{"foo":["baz"]}

The connection stays open and results keep streaming in. To check it out in a client, you can do:

curl --raw http://gweetapp.appspot.com/stream/test/?streaming=1

or, with Python and requests:

import requests
r = requests.get("http://gweetapp.appspot.com/stream/test/?streaming=1", stream=True)

for line in r.iter_lines(1):
    print(line)

The script will just never terminate and you’ll get the lines as they come in. Make sure to ignore empty lines, they’re keepalives (once every 30s). Data will arrive immediately as another client POSTs it. Try testing it out yourself, it’s really really fun!

Live demo

I published a sample, fully-functional instance of the server on AppEngine, so you can play with it. Use the API above to make requests to:

http://gweetapp.appspot.com/

The server itself is pretty high-performance, able to service around 10,000 requests per second, and the AppEngine instance is free for use, as long as it doesn’t get abused. If any miscreants give it any trouble, it may go down or get removed at any time, so don’t rely on it too much.

Let me know if you have any feedback, and pull requests are much appreciated!