Stavros' Stuff

This post is going to be short, but hopefully will help you avoid the troubles that befell me. I wanted to make a Slack bot using Python. “How hard can it be?”, I thought. “I’ve done it many times before”, I thought.

Think again.

The problem is that Slack has changed the way their APIs work. The old way is now referred to as a “classic app” with a “bot scope”, and that way is deprecated and you can’t really create apps like that now, so you have to do a whole other thing.

In this post, I will detail the steps necessary to create a simple bot that will listen for messages and reply to them. That’s all the scaffolding you’ll need (or that I needed) to create your apps, but I had to search for many hours to discover this information. Hopefully Google will be kinder to you and

Continue reading…

You know me, I’m a Django fan. It’s my preferred way of developing web apps, mainly because of the absolutely vast ecosystem of apps and libraries it has, and the fact that it is really well-designed. I love how modular it is, and how it lets you use any of the parts you like and forget about the ones you don’t want. This is going to be emphasized rather spectacularly in this article, as I’m going to do things nobody should ever have to do.

My only issue with Django was that it never really had a good way of making APIs. I hate DRF with somewhat of a passion, I always found its API way too complicated and verbose, and never managed to grok it. Even the simplest things felt cumbersome, and the moment your API objects deviated from looking exactly like your DB models, you were in a world of hurt. I generally prefer writing a simple class-based view for my APIs, but then I don’t get automatic docs and other niceties.

It’s no surprise, then, that when I found FastAPI I was really excited, I really liked its autogenerated docs, dependency injection system, and lack of magical “request” objects or big JSON blobs. It looked very simple, well-architected and with sane defaults, and I seriously considered developing the API for my company’s next product on it, but was apprehensive about two things: It lacked Django’s ecosystem, and it didn’t have an ORM as good and well-integrated as Django’s. I would also miss Django’s admin interface a lot. Three things.

It would have been great if FastAPI was a Django library, but I guess the asynchronicity wouldn’t have been possible. Still, there’s no reason for DRF not to have an API as nice as FastAPI’s, but there’s no helping that. A fantastical notion caught hold of me: What if I could combine FastAPI’s view serving with Django’s ORM and apps? Verily, I say unto thee, it would be rad.

And that’s exactly what I did. Here’s how:

Continue reading…

Ever since I was a wide-eyed little boy, I would look up at the stars and wonder in wonder: “What if I could lease my very own, beefy, dedicated Hetzner server and have an easy way to deploy all my projects onto that?” But lo, my dreams were dashed because Docker wouldn’t be invented for another twenty years, and Hetzner did not accept Mastercard at the time.

Decades later, with Docker finally invented and Hetzner accepting all major credit cards, my dream lay all but forgotten, because Docker could not do zero-downtime deploys natively and I hated it. That was how things remained, until my friend Theodore told me that he tried Dokku and that it worked very well.

I had heard of Dokku (and Fig, Deis, Flynn, Kubernetes, etc etc), but I never paid too much attention, as these PaaSaaSes struck me as too webcale for my simple projects. All I wanted was a way to skip through all the boilerplate configuration of deploying a Django app, and Ansible wasn’t cutting it, as it was still too much plumbing.

Since Theodore tried it and said it was apparently pretty easy to deploy with, though, I figured I’d give it a shot and see. It helped that Dokku was explicitly designed to be light and self-contained, whereas Kubernetes is for much larger deployments, so Dokku fit my use case exactly.

Trying Dokku out

To try Dokku out, I needed a project. Luckily,

Continue reading…

I finally managed to deploy Django in a Docker container on production! I’ve been trying to switch to a full Docker development/production model since Docker came out, but only recently did the ecosystem mature enough to allow me to easily use Docker both for development (where it excels, in my opinion) and on production (where it’s pretty okay and quite useful).

In this post, I will quickly give you all the relevant details and files you need to go from a newly checked-out repository to a full development environment in one command, and to deploy that service to production. As a bonus, I’ll show you how to use Gitlab (which is awesome) to build your containers and store them in the Gitlab registry.

Let’s begin!

Development

Continue reading…

After designing my first PCB, I went on a designing spree. It turns out that making PCBs (printed circuit boards, basically a piece of plastic that includes all the connections of your components in it. It helps make your project smaller and cut down on the amount of wires floating around) is so enjoyable, I’m PCBing all the things! The next victim for PCBfication is a circuit I had originally built on an Arduino and subsequently migrated to an ESP8266.

The circuit is a home sensor and controller. It can sense light, temperature, humidity and motion, and includes an RF controller (at 433 MHz) and an infrared LED so you can control your TV and other home devices. In this post, I’ll go into some detail about the build and how it connects to other sensors and controllers around the house.

This post is also a test of my new Expounder concept library. Throughout the post, various terms will be underlined like this (with a dashed underline), and you can click on them if you’re unfamiliar with the underlined term. After clicking, some text will expand and explain the term.

Let’s continue to

Continue reading…

As you can probably tell from previous posts, I’ve been pretty into hardware lately. I’ve especially been building things like home sensors and controllers, so I have a central computer reading motion, temperature, humidity, light and other values in the house and deciding whether the lights or air conditioning need to be on or off.

I also want to be able to turn these on and off from my mobile phone, from anywhere in the world. The problem with that is that I need a way to ensure that only my phone can turn things on in my house. I wouldn’t want someone to be able to turn the heating on in my house at full blast when I’m not there and waste all my electricity bill (or set fire to something).

TLS is a pretty good solution, as it ensures confidentiality between client and server, but it does nothing for verifying the client or securing communications against a malicious server. I needed something better, and I couldn’t find something readily available. So I set out to write it. Thus, string phone was born.

String phone is a

Continue reading…

I very recently discovered the very useful pychromecast library, and, being the owner of a Chromecast, I started wondering what I could do with it. Since there aren’t that many things a Chromecast can do (it pretty much only plays videos over the network, although there are some other things as well), I decided to make my own TV station. Thus, Shufflecast was born.

Continue reading…

Earlier today, i.e. a few minutes ago, I was working on my latest guinea pig, TiThess. It’s an events guide for my city, and the latest project I’m trying everything on. I like having a project I can try new things on, as it helps keep my skills sharp.

As I was testing page load times with the excellent Web Page Test, trying to get them down to the absolute minimum, I was getting an F in time-to-first-byte. This is very odd, because the whole site is supposed to be cached, so I was wondering whether the cache is doing something wrong and slowing page generation down.

To make sure, I needed a simple way to show how long page generation took, like the old “page generated in X seconds” footer that was all the rage with PHP sites way back when. Here’s how I did it:

Continue reading…

This post needs no introduction, so it doesn’t have one. If you want to write an API and need to know how to make it secure, and have the requests authenticate against a server or a client, look no further! Well, do look a bit further, because I’m going to tell you how to do all these things in this post.

Use cases

As with most other things, your API authentication method will depend on your use case. I will detail a few common ones, along with the best authentication scheme for each one:

Continue reading…

You know the deal, you have your fantastic Django application and it’s working great and everything, but you need to make a small change which is too cumbersome to do in the shell, so you figure “duh, I’ll just write a script to do it”. You write your external script in two minutes and then struggle for two hours to figure out how to load the models and the rest of the context so it will work with your app’s settings and all your Django goodness.

You visit StackOverflow and a bunch more sites, and they either tell you to use a management command, which is great advice, except your thing is a one-off and you don’t want to have to check it to git and go through all that hassle to get it deployed just to do this simple thing, or they give you some arcane lines that just don’t work.

Fear not, for I am here. I will give you five simple lines that will make everything work perfectly. Perfectly, I say!

Without further ado (all the previous ado was just so I could fill the paragraph so the side-box doesn’t look weird with short text), I give you the magic commands! Here they are:

Continue reading…

One of my pet peeves when it comes to Django is that you can’t call methods that require arguments in templates. While this is fine most of the time, it does mean that you need to have one property or method per call you want to make, which sometimes gets very cumbersome.

I needed a way to define various dynamic permissions that are calculated at runtime (for irrelevant reasons, Django’s permissions framework wasn’t a good fit), and writing properties like can_register, can_add_tags, can_subscribe got tedious. These tended to be defined all over the place, rather than in one central spot, and it was hard to add more checks without cluttering the classes.

I would much prefer to have a single method (let’s call it can()) that accepted a string with the permission I wanted to check, and return True or False, depending. This is easy to do in the views, but templates would never be able to call it with an argument.

However, since Django can do dictionary-style attribute lookups, I could add a dictionary interface over the method, and allow

Continue reading…

If you’re a regular reader, you might have noticed that I’ve been on a quest for the perfect backup program, and ended up writing my own encryption layer over bup.

While writing encbup, I wasn’t very satisfied with having to download the entire huge archive just to restore a file, and still wished that I could use EncFS together with rdiff-backup to have true remote-mountable, encrypted, deduplicated, versioned backups.

Trying obnam again (spoiler: it’s still pretty slow), I noticed that it included a mount command. Looking at it, I discovered fuse-python and fusepy, and realized that writing a FUSE filesystem in Python is pretty much trivial.

The astute observer will have already realized where I’m going with this: I decided to write an encrypted filesystem layer in Python! This layer would be very similar to EncFS, with a few crucial differences:

Continue reading…

Note: Be sure to check the sequel to this post, about the program that will supersede this one and be compatible with all backup utilities.

Backing things up is important, and, luckily, there are many high-quality services geared to everyday people that are very easy to use and cheap. Unfortunately, I am not everyday people, as I am very paranoid and insist that absolutely nobody be able to see my photos of my dog and lawn. It’s a matter of privacy.

To that end, I’ve long been looking for a secure/encrypted backups service, but I haven’t managed to find a single service or tool that fulfils my requirements:

• Cheap to store data on (~$30 per year for

Continue reading…

A few days ago, I wrote a post about a peculiar piece of code that a friend of mine had sent me. Since it was interesting bit of code, I thought Hacker News would enjoy it, so I posted it there. To my great pleasure, the post shot up to the first place in a few minutes and continued there for a full day, bringing just over 50,000 visitors to this blog, in total.

I was very happy that people were liking and discussing this post (and the discussion was very interesting in its own right), but I noticed that AppEngine, where this blog is hosted, was struggling to serve it. I had to create new instances because the average latency was about ten seconds(!), even though this blog is pretty much only text and static media, and I use Django’s per-site cache to cache every single page.

Continue reading…

I came upon a very interesting and cryptic snippet of code somewhere nameless, and I can’t decide if it is brilliant or completely insane. It is a very obscure way of accomplishing the required task, but it’s around four times faster than the alternatives I’ve tried, so I have to admit that it’s not completely without merit. Still, I cringe a bit at seeing it, since it packs around four unusual Python concepts in almost as many characters.

This is the snippet in question:

Continue reading…

Today’s tip is brought to you by the color green and the number g.

If you’re an industrious Python programmer, you probably use pip and virtualenv to install your packages in an isolated location, which is great, and good job to you. However, having to install Django, pandas and ipython every time you want to experiment with something is a bit tiresome, since they’re all big files and downloading them takes a while, plus what if you’re on a data-limited connection and bla bla bla.

Here’s a very easy way to both speed up your downloads and avoid having to do them in the first place:

Continue reading…

If you haven’t heard yet, let me break the good news to you: Django 1.6 is getting template apps/projects. This means that, when starting a new project, you don’t have to copy your old directories and change stuff around any more, you can just put them into reusable directories (or even repositories) and Django will be able to use them very simply:

django-admin.py startproject --template=/djtemplates/default myapp

I’ve been needing a feature like that for ages, and now I’m giddy with excitement, so, without further ado,

Continue reading…

I have suddenly noticed that my computer’s fans have been more noisy than when I first bought it, and now it’s driving me crazy. I wondered if I could cut the fans’ power without increasing the temperatures inside the case too much, or even leave the case open. To test it, I spent five minutes writing a small script to monitor temperatures in Python, using smartmontools and lm-sensors.

This is that script:

Continue reading…

If you use Django for your web apps, you will probably have come across a situation where you need to run some custom code on assigning a value to a model field. You’ve probably hacked around this by overriding the save() method or some other arcane hackery, but this is not only unnecessary, but it’s more complicated than it should be.

You can use a Python [property](http://docs.python.org/2/libr

Continue reading…

I’ve recently been looking at syntax highlighters, both for this blog and for Instahero, the new product my company is developing. I used to use a JS-based solution for the blog, but it broke with many inputs, it was fiddly to work with, didn’t parse correctly sometimes, and slowed the site down considerably. I especially disliked the fact that you had to include one JS

Continue reading…

If you’ve ever programmed in Python, you have a 96.2% chance of having used the range() function. As you have a 96.1% chance of knowing, said function returns a list containing an arithmetic progression of integers, exactly as the documentation says. For example, range(4) returns [0, 1 2, 3].

If you’ve ever bothered to look at the docstring, you will see that it goes thus:

Continue reading…

I’ve been hard at work these days, and to relax I decided to develop a small application/side project. As with many applications, I needed a reliable, persistent store, so I went with the most reliable, easy to set up, performant, and generally awesome store I knew: SQLite.

However, since this was a very experimental prototype, I got tired of the frequent schema changes and wished I had something

Continue reading…

As I’ve mentioned many times on this blog, I’m a big fan of Google AppEngine. I like the free tier, the effortless scaling, the zero administration, and how I can run Django almost unmodified and deploy in seconds. It has its downsides, but overall I like it very much.

The one part of deployment I didn’t like was dependency handling. Due to the way GAE works, you need to install all packages in t

Continue reading…

Often I find that I need to include administration views in my site, such as statistics, management, etc etc. When these are more than just model CRUD views, a good idea is to extend the builtin Django admin site, as it’s the easiest to do.

While looking at the docs, however, I realised they were a bit cryptic and lacked good examples. It didn’t take me long to figure out how to do it, but here’s

Continue reading…

Well, I have a computer architecture exam in six hours and can’t be bothered, so I figured I would realize a lifelong dream of mine, and make a program that prints “Hello world!” using curve fitting techniques. Enlisting the help of a good friend with numerous mathematical papers under his belt (ostensibly because he could not afford a tighter belt), MATLAB and a longing for procrastination, we em

Continue reading…