Stavros' Stuff Latest Posts Latest posts on Stavros' Stuff. en-us Stavros Korokithakis How to use FIDO2 USB authenticators with SSH https://www.stavros.io/posts/u2f-fido2-with-ssh/ https://www.stavros.io/posts/u2f-fido2-with-ssh/ <div class="pull-quote">Secure, easy to use, cheap: Pick three</div><p>I recently installed Ubuntu Wacky Whatever, the latest version, and I&#8217;m very excited about it shipping with SSH 8.2, which means that I can finally use hardware USB keys for secure, easy to use authentication. If securing your devices has been something you&#8217;ve wanted to easily do yourself, read on, because it&#8217;s finally happening.</p> <h2>FIDO2</h2> <p>One of the most exciting security-related developments recently has been the development of <a href="https://webauthn.guide/">WebAuthn</a> and FIDO2, which are basically euphemisms for &#8220;nice security stuff&#8221;. In summary, WebAuthn and FIDO2 aim to make it really easy to use security devices with stuff by standardizing the way the two talk to each other, and using better terms than &#8220;stuff&#8221;.</p> <p>This is great news for us, because now we can have dirt-cheap USB keys that can be used to secure all our authentication very easily, without requiring any special security knowledge. All you need to know to be completely immune to phishing, password theft, and a whole host of other ways of losing Bitcoin is to just plug your USB key in, press the little button/type your PIN/enter your fingerprint, and you&#8217;re logged in.</p> <p>What does this have to do with SSH? Very little, but<!-- break --> I like rambling, it is known.</p> <h3>Using FIDO2 with SSH</h3> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><img src="shoulder-small.jpg"><a class="photocredit" href="https://unsplash.com/photos/OLzlXZm_mOw" title="Photo from https://unsplash.com/photos/OLzlXZm_mOw" target="_blank" rel="noopener noreferrer"><i class="fa fa-camera"></i></a></div><span class="caption">Deanna had heard of phishing, and was extremely suspicious of the sound of rubber boots.</span></div><p>As you may have suspected from the two seemingly unrelated points above, and the dead giveaway title of this section, it would be tremendous if we could use FIDO2 keys to authenticate over SSH, and the OpenSSH developers made it happen. OpenSSH now supports FIDO (U2F) and FIDO2 for authentication, with FIDO2 with resident keys having a nicer UX.</p> <p>At a high level, the USB device (FIDO2 devices don&#8217;t have to be USB, but they usually will be) will generate a <span data-expounder="secret-key">secret key</span>. <span data-expounded="secret-key"> The device can then use this secret key to prove that you have the device in your possession, without the key ever leaving the device. That means that, by extension, you can use the device to prove you are who you say you are to any server who asks. </span> The key is either stored on the device (called &#8220;resident key&#8221;), which requires the device to support storage, or it isn&#8217;t, and <span data-expounder="storage">you need to store the data somewhere else</span>.</p> <p><span data-expounded="storage"> The way this works, in my understanding (which I&#8217;m not certain about and please correct me if I&#8217;m wrong), is the following: Each device comes from the factory with one key pair/seed built in, but it can accept an additional seed to derive more pairs from, by combining its current seed with the one given. This allows devices to generate keys without having any storage space, but it does require you to be giving them the seed every time. Devices with &#8220;resident key&#8221; capabilities can store the whole seed on their memory, and don&#8217;t need you to do anything else. </span></p> <p>The benefit of using a device like this including eliminating phishing, password theft, authentication replay, and lots of other attacks. Since the device authenticates to a specfic realm (server address/URL/etc), an attacker can&#8217;t reuse one site&#8217;s authentication on another site, which makes phishing impossible. Nobody can steal your password or private key either, since it&#8217;s on the device itself and extracting it is impossible. The only plausible attack is physical theft with the device, which is mitigated by making the device ask for a PIN or fingerprint and wiping itself after a few wrong attempts.</p> <p>Another benefit with having this built into SSH is that you don&#8217;t specifically need a Yubikey, or to mess with extra software like Yubikey agent, PIV mode or anything else. You just plug any FIDO2-compatible key in, and you can use it with SSH. I tried the following on both my Yubikey 5C, Yubikey FIDO2 and Yubikey NEO, they worked like a charm on all of them, but &#8220;resident key&#8221; mode only works on the 5C and later versions of the FIDO2 key.</p> <h2>Actual usage</h2> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><img src="padlocks-small.jpg"><a class="photocredit" href="https://unsplash.com/photos/bBavss4ZQcA" title="Photo from https://unsplash.com/photos/bBavss4ZQcA" target="_blank" rel="noopener noreferrer"><i class="fa fa-camera"></i></a></div><span class="caption">With twelve padlocks, the city of Indianapolis was sure it had put a stop to miscreants stealing its favorite fence.</span></div><p>After all the needless detail about how devices work, we&#8217;re ready to actually use one for SSH. Setting that up is really simple, basically one or two commands (depending on the method) with no extra software to install, though you will need to be running OpenSSH 8.2 on both client and server.</p> <p>I will split the instructions into two parts, one for if your key supports (and you want to use) the resident key mode, and one for if it doesn&#8217;t. There&#8217;s no harm in using either method, so if you don&#8217;t know if your device supports resident key mode, just try the first method and continue to the second one if it fails.</p> <h3>Resident key mode</h3> <p>Use the following command to generate a key and store it on the device:</p> <div class="highlight"><pre><span></span>ssh-keygen -t ed25519-sk -O resident -f ~/.ssh/id_mykey_sk </pre></div> <p>SSH will ask you to enter your PIN and touch your device, and then save the key pair where you told it. If that command complains about ed25519 not being available, try this one:</p> <div class="highlight"><pre><span></span>ssh-keygen -t ecdsa-sk -O resident -f ~/.ssh/id_mykey_sk </pre></div> <p>OpenSSH will save two files, one called <code>id_mykey_sk</code>, and one called <code>id_mykey_sk.pub</code>. You only really need the latter, it&#8217;s your public key, and you can now add it to the <code>authorized_keys</code> file of any server that runs OpenSSH 8.2 and above.</p> <p>To add your private key to the SSH agent, you can either copy <code>id_mykey_sk</code> to your <code>~/.ssh/</code> directory, or (if you&#8217;re at a new computer and don&#8217;t have the key with you), run</p> <div class="highlight"><pre><span></span>ssh-add -K </pre></div> <p>with your device plugged in. This will add the key to the SSH agent, and you will be able to connect to servers immediately.</p> <p>To get your public key from the SSH agent, run:</p> <div class="highlight"><pre><span></span>ssh-add -L </pre></div> <p>This will show you a list of public keys, including the FIDO2 one.</p> <h3>Non-resident key mode</h3> <p>The non-resident key mode is the same as the previous mode, except you can&#8217;t load your key with <code>ssh-add -K</code> directly from the device. As before, run:</p> <div class="highlight"><pre><span></span>ssh-keygen -t ed25519-sk -f ~/.ssh/id_mykey_sk </pre></div> <p>SSH will ask you to enter your PIN and touch your device, and then save the key pair where you told it. If that command complains about ed25519 not being available, try this one:</p> <div class="highlight"><pre><span></span>ssh-keygen -t ecdsa-sk -f ~/.ssh/id_mykey_sk </pre></div> <p>OpenSSH will save two files, one called <code>id_mykey_sk</code>, and one called <code>id_mykey_sk.pub</code>. You need both of these files, store <code>id_mykey_sk</code> in your <code>~/.ssh/</code> directory (where it got generated) and add the <code>id_mykey_sk.pub</code> file to any server you want to log in to.</p> <p>That&#8217;s it, when you try to log in, OpenSSH will ask you for your PIN and to touch the device.</p> <h2>Security considerations</h2> <p>You can pass the <code>no-touch-required</code> option to <code>ssh-keygen</code> to tell it that you don&#8217;t want it to require touching the device every time, but I haven&#8217;t had great results with enabling that. If you try it, let me know how well it works in the comments, but it&#8217;s probably better for security if you leave it disabled.</p> <p>Storing your key on the device means that nobody will be able to steal it (hopefully your device has a PIN or other authentication method set), but untrusted computers might still be able to send commands over the SSH session after you connect. Don&#8217;t SSH from computers you don&#8217;t trust, and be extra careful when using SSH forwarding (the <code>-A</code> option), as the server can then ask your computer to authenticate to other servers on your behalf.</p> <h2>Epilogue</h2> <p>I&#8217;m very excited about FIDO2 in general, and for SSH authentication specifically, because we finally have a way to secure authentication that my parents can easily use, with a much smaller attack surface. I&#8217;m particularly interested in seeing how WebAuthn takes off for authenticating on the web without usernames and passwords (see my <a href="https://gitlab.com/stavros/django-webauthin">django-webauthin</a> library for a demo), and hope it becomes widespread.</p> <p>If you have any feedback, please <a href="https://twitter.com/intent/user?screen_name=Stavros">Tweet</a> or <a href="https://mastodon.social/@stavros">toot</a> at me, or email me directly. Also, check out my <a href="https://www.youtube.com/c/StavrosKorokithakis/">YouTube channel</a> (because who <em>doesn&#8217;t</em> have a YouTube channel these days?), where I frequently livestream my late-night maker sessions.</p> Tue, 30 Jun 2020 00:30:19 +0000 Using FastAPI with Django https://www.stavros.io/posts/fastapi-with-django/ https://www.stavros.io/posts/fastapi-with-django/ <div class="pull-quote">FastAPI actually plays very well with Django</div><p>You know me, I&#8217;m a <a href="https://www.djangoproject.com/">Django</a> fan. It&#8217;s my preferred way of developing web apps, mainly because of the absolutely <em>vast</em> ecosystem of apps and libraries it has, and the fact that it is really well-designed. I love how modular it is, and how it lets you use any of the parts you like and forget about the ones you don&#8217;t want. This is going to be emphasized rather spectacularly in this article, as I&#8217;m going to do things nobody should ever have to do.</p> <p>My only issue with Django was that it never really had a good way of making APIs. I hate DRF with somewhat of a passion, I always found its API way too complicated and verbose, and never managed to grok it. Even the simplest things felt cumbersome, and the moment your API objects deviated from looking exactly like your DB models, you were in a world of hurt. I generally prefer writing a simple class-based view for my APIs, but then I don&#8217;t get automatic docs and other niceties.</p> <p>It&#8217;s no surprise, then, that when I found <a href="https://fastapi.tiangolo.com/">FastAPI</a> I was really excited, I really liked its autogenerated docs, dependency injection system, and lack of magical &#8220;request&#8221; objects or big JSON blobs. It looked very simple, well-architected and with sane defaults, and I seriously considered developing the API for my company&#8217;s next product on it, but was apprehensive about two things: It lacked Django&#8217;s ecosystem, and it didn&#8217;t have an ORM as good and well-integrated as Django&#8217;s. I would also miss Django&#8217;s admin interface a lot. Three things.</p> <p>It would have been great if FastAPI was a Django library, but I guess the asynchronicity wouldn&#8217;t have been possible. Still, there&#8217;s no reason for DRF not to have an API as nice as FastAPI&#8217;s, but there&#8217;s no helping that. A fantastical notion caught hold of me: What if I could combine FastAPI&#8217;s view serving with Django&#8217;s ORM and apps? Verily, I say unto thee, it would be rad.</p> <p>And that&#8217;s exactly what I did. Here&#8217;s how:</p> <!-- break --> <h2>General details</h2> <p>Each part of this unholy union needed a different integration. I will go into details on each part here, and post code in the next section.</p> <h3>High-level overview</h3> <p>The way this profane joining works is by using FastAPI as the view layer, and importing and using parts of Django separately. This means that some things like middleware will obviously not work, since Django is not handling views at all.</p> <p>I also didn&#8217;t try to get asynchronicity working, which might be a dealbreaker for many people. I suspect I&#8217;d have issues, as Django doesn&#8217;t support it very well yet, and the ORM would probably block all view threads. If you do get it working, or try to, please leave a comment here or contact me.</p> <h3>ORM</h3> <p>The ORM works quite well, basically exactly as if you were using Django. Migrations work fine, and any app that interfaces with or enhances the ORM should also work properly.</p> <h3>Tests</h3> <p>Tests are working great as well, using Django&#8217;s <code>./manage.py test</code> harness/discovery. If you want to use another runner like pytest, that should also work properly, though I haven&#8217;t tested it.</p> <p>To clarify, we&#8217;ll be testing the <em>FastAPI</em> views using Django&#8217;s test runner and <code>./manage.py test</code>. The one problem I had was that the usual <code>TestCase</code> class didn&#8217;t work (it gave me &#8220;table is locked&#8221; errors), but <code>TransactionTestCase</code> works splendidly and is probably quite a bit faster, too.</p> <h3>The admin and other views</h3> <p>The admin is a vital part of Django, it makes dealing with your models much, much easier than having to wrangle with the database, so it was important to me to get working. And I did! It&#8217;s a pretty hacky way, but it works perfectly, since the trick is to just run Django on another port and have it serve the admin. You can have it serve any other views you want, this way you can use FastAPI to serve your API and use Django for static files and everything else (assuming you have anything else).</p> <p>All you need is a second stanza in your reverse proxy.</p> <h2>The code</h2> <p>Without further delay, the part you&#8217;ve all been waiting for, the code. This section includes both the bare essentials for getting the integration working, and some best practices I&#8217;ve found when working with FastAPI, such as URL reversing, easier DB model to API model conversion, etc. I hope you&#8217;ll find it useful.</p> <h3>Starting out</h3> <p>To begin, install FastAPI, Uvicorn and Django and create a Django project as you normally would. I recommend my <a href="https://github.com/skorokithakis/django-project-template">&#8220;new project&#8221; template</a>, it contains various niceties you may want to use. We&#8217;re going to replace some parts and add others. I&#8217;ve created a project called <code>goatfish</code> and an app called <code>main</code> to illustrate, you can call yours whatever you want. Make sure you add <code>"main"</code> to Django&#8217;s <code>INSTALLED_APPS</code> list in the settings so it discovers the models and migrations.</p> <h3>The models</h3> <p>We should probably start with our data model so everything else makes sense. The API is going to be a straightforward CRUD API, which will serve a model we&#8217;ll call <code>Simulation</code> and provide authentication.</p> <p>Since we have an API and a database, and the models for the two are neither semantically nor functionally identical, we&#8217;ll have two sections in our <code>models.py</code>, one for each model type.</p> <p><strong><code>main/models.py</code></strong>:</p> <div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span> <span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Dict</span> <span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span> <span class="kn">import</span> <span class="nn">shortuuid</span> <span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="kn">import</span> <span class="n">AbstractUser</span> <span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">models</span> <span class="kn">from</span> <span class="nn">pydantic</span> <span class="kn">import</span> <span class="n">BaseModel</span> <span class="k">def</span> <span class="nf">generate_uuid</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span> <span class="sd">&quot;&quot;&quot;Generate a UUID.&quot;&quot;&quot;</span> <span class="k">return</span> <span class="n">shortuuid</span><span class="o">.</span><span class="n">ShortUUID</span><span class="p">()</span><span class="o">.</span><span class="n">random</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span> <span class="c1">##########</span> <span class="c1"># Models</span> <span class="c1">#</span> <span class="k">class</span> <span class="nc">CharIDModel</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;Base model that gives children random string UUIDs.&quot;&quot;&quot;</span> <span class="nb">id</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span> <span class="n">max_length</span><span class="o">=</span><span class="mi">30</span><span class="p">,</span> <span class="n">primary_key</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="n">generate_uuid</span><span class="p">,</span> <span class="n">editable</span><span class="o">=</span><span class="kc">False</span> <span class="p">)</span> <span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span> <span class="n">abstract</span> <span class="o">=</span> <span class="kc">True</span> <span class="k">class</span> <span class="nc">User</span><span class="p">(</span><span class="n">AbstractUser</span><span class="p">):</span> <span class="n">api_key</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span> <span class="n">max_length</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="n">generate_uuid</span><span class="p">,</span> <span class="n">db_index</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="p">)</span> <span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">username</span> <span class="k">class</span> <span class="nc">Simulation</span><span class="p">(</span><span class="n">CharIDModel</span><span class="p">):</span> <span class="n">user</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="n">User</span><span class="p">,</span> <span class="n">on_delete</span><span class="o">=</span><span class="n">models</span><span class="o">.</span><span class="n">CASCADE</span><span class="p">)</span> <span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">300</span><span class="p">)</span> <span class="n">start_date</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">DateField</span><span class="p">()</span> <span class="n">end_date</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">DateField</span><span class="p">()</span> <span class="nd">@classmethod</span> <span class="k">def</span> <span class="nf">from_api</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">user</span><span class="p">:</span> <span class="n">User</span><span class="p">,</span> <span class="n">model</span><span class="p">:</span> <span class="s2">&quot;DamnFastAPISimulation&quot;</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Return a Simulation instance from an APISimulation instance.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">return</span> <span class="bp">cls</span><span class="p">(</span> <span class="n">user</span><span class="o">=</span><span class="n">user</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="n">model</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">model</span><span class="o">.</span><span class="n">start_date</span><span class="p">,</span> <span class="n">end_date</span><span class="o">=</span><span class="n">model</span><span class="o">.</span><span class="n">end_date</span><span class="p">,</span> <span class="p">)</span> <span class="k">def</span> <span class="nf">update_from_api</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">api_model</span><span class="p">:</span> <span class="s2">&quot;DamnFastAPISimulation&quot;</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Update the Simulation Django model from an APISimulation instance.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">api_model</span><span class="o">.</span><span class="n">name</span> <span class="bp">self</span><span class="o">.</span><span class="n">start_date</span> <span class="o">=</span> <span class="n">api_model</span><span class="o">.</span><span class="n">start_date</span> <span class="bp">self</span><span class="o">.</span><span class="n">end_date</span> <span class="o">=</span> <span class="n">api_model</span><span class="o">.</span><span class="n">end_date</span> <span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="c1">##########</span> <span class="c1"># Types</span> <span class="c1">#</span> <span class="c1"># Over here I&#39;m really ticked off I need to make two models just to</span> <span class="c1"># exclude the id field from being required in the bodies of POSTs.</span> <span class="c1">#</span> <span class="c1"># Please comment on this bug if you feel as strongly:</span> <span class="c1"># https://github.com/tiangolo/fastapi/issues/1357</span> <span class="k">class</span> <span class="nc">DamnFastAPISimulation</span><span class="p">(</span><span class="n">BaseModel</span><span class="p">):</span> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span> <span class="n">start_date</span><span class="p">:</span> <span class="n">date</span> <span class="n">end_date</span><span class="p">:</span> <span class="n">date</span> <span class="nd">@classmethod</span> <span class="k">def</span> <span class="nf">from_model</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">instance</span><span class="p">:</span> <span class="n">Simulation</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Convert a Django Simulation model instance to an APISimulation instance.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">return</span> <span class="bp">cls</span><span class="p">(</span> <span class="nb">id</span><span class="o">=</span><span class="n">instance</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="n">instance</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">instance</span><span class="o">.</span><span class="n">start_date</span><span class="p">,</span> <span class="n">end_date</span><span class="o">=</span><span class="n">instance</span><span class="o">.</span><span class="n">end_date</span><span class="p">,</span> <span class="p">)</span> <span class="k">class</span> <span class="nc">APISimulation</span><span class="p">(</span><span class="n">DamnFastAPISimulation</span><span class="p">):</span> <span class="nb">id</span><span class="p">:</span> <span class="nb">str</span> <span class="k">class</span> <span class="nc">APISimulations</span><span class="p">(</span><span class="n">BaseModel</span><span class="p">):</span> <span class="n">items</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="n">APISimulation</span><span class="p">]</span> <span class="nd">@classmethod</span> <span class="k">def</span> <span class="nf">from_qs</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">qs</span><span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Convert a Django Simulation queryset to APISimulation instances.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">return</span> <span class="bp">cls</span><span class="p">(</span><span class="n">items</span><span class="o">=</span><span class="p">[</span><span class="n">APISimulation</span><span class="o">.</span><span class="n">from_model</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">qs</span><span class="p">])</span> </pre></div> <p>A few notes: Each class instance has helper methods, like <code>from_api</code>, <code>from_model</code>, <code>from_qs</code>, etc to facilitate converting between API-level and DB-level objects easily.</p> <p>Also, unfortunately we have to make two separate type classes just to avoid having the <code>id</code> field show up in the <code>POST</code> request, as the user of the API should not be able to send/set the id when creating a new object. It&#8217;s annoying, and I have opened an issue on <a href="https://github.com/tiangolo/fastapi/issues/1357">FastAPI&#8217;s GitHub</a>.</p> <h3>The entry point</h3> <p>To serve requests, we need a place for our wsgi app to live. The best place is, naturally, <code>wsgi.py</code>:</p> <p><strong><code>goatfish/wsgi.py</code></strong>:</p> <div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span> <span class="kn">from</span> <span class="nn">django.core.wsgi</span> <span class="kn">import</span> <span class="n">get_wsgi_application</span> <span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">FastAPI</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="s2">&quot;DJANGO_SETTINGS_MODULE&quot;</span><span class="p">,</span> <span class="s2">&quot;goatfish.settings&quot;</span><span class="p">)</span> <span class="n">application</span> <span class="o">=</span> <span class="n">get_wsgi_application</span><span class="p">()</span> <span class="kn">from</span> <span class="nn">main.urls</span> <span class="kn">import</span> <span class="n">router</span> <span class="k">as</span> <span class="n">main_router</span> <span class="n">app</span> <span class="o">=</span> <span class="n">FastAPI</span><span class="p">(</span> <span class="n">title</span><span class="o">=</span><span class="s2">&quot;Goatfish&quot;</span><span class="p">,</span> <span class="n">description</span><span class="o">=</span><span class="s2">&quot;A demo project. Also, an actual fish with a weird name.&quot;</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s2">&quot;We aren&#39;t doing versions yet. Point oh.&quot;</span><span class="p">,</span> <span class="p">)</span> <span class="n">app</span><span class="o">.</span><span class="n">include_router</span><span class="p">(</span><span class="n">main_router</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s2">&quot;/api&quot;</span><span class="p">)</span> </pre></div> <p>This will allow Uvicorn to load and serve our app. Keep in mind that <code>app</code> is FastAPI&#8217;s entry point and <code>application</code> is Django&#8217;s. You&#8217;ll need to remember that if you deploy them to something that needs to load the WSGI entry points.</p> <p>You can start the server by running <code>uvicorn goatfish.wsgi:app --reload</code>, but it&#8217;ll probably crash at this point because there are no URL routes yet. Let&#8217;s add them.</p> <h3>The routes</h3> <p>I prefer the Django convention of having each app&#8217;s route in a <code>urls.py</code> in the app directory, so let&#8217;s put them there:</p> <p><strong><code>main/urls.py</code></strong>:</p> <div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">APIRouter</span> <span class="kn">from</span> <span class="nn">main</span> <span class="kn">import</span> <span class="n">views</span> <span class="c1"># The API model for one object.</span> <span class="kn">from</span> <span class="nn">main.models</span> <span class="kn">import</span> <span class="n">APISimulation</span> <span class="c1"># The API model for a collection of objects.</span> <span class="kn">from</span> <span class="nn">main.models</span> <span class="kn">import</span> <span class="n">APISimulations</span> <span class="n">router</span> <span class="o">=</span> <span class="n">APIRouter</span><span class="p">()</span> <span class="n">router</span><span class="o">.</span><span class="n">get</span><span class="p">(</span> <span class="s2">&quot;/simulation/&quot;</span><span class="p">,</span> <span class="n">summary</span><span class="o">=</span><span class="s2">&quot;Retrieve a list of all the simulations.&quot;</span><span class="p">,</span> <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;simulations&quot;</span><span class="p">],</span> <span class="n">response_model</span><span class="o">=</span><span class="n">APISimulations</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">&quot;simulations-get&quot;</span><span class="p">,</span> <span class="p">)(</span><span class="n">views</span><span class="o">.</span><span class="n">simulations_get</span><span class="p">)</span> <span class="n">router</span><span class="o">.</span><span class="n">post</span><span class="p">(</span> <span class="s2">&quot;/simulation/&quot;</span><span class="p">,</span> <span class="n">summary</span><span class="o">=</span><span class="s2">&quot;Create a new simulation.&quot;</span><span class="p">,</span> <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;simulations&quot;</span><span class="p">],</span> <span class="n">response_model</span><span class="o">=</span><span class="n">APISimulation</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">&quot;simulations-post&quot;</span><span class="p">,</span> <span class="p">)(</span><span class="n">views</span><span class="o">.</span><span class="n">simulation_post</span><span class="p">)</span> <span class="n">router</span><span class="o">.</span><span class="n">get</span><span class="p">(</span> <span class="s2">&quot;/simulation/</span><span class="si">{simulation_id}</span><span class="s2">/&quot;</span><span class="p">,</span> <span class="n">summary</span><span class="o">=</span><span class="s2">&quot;Retrieve a specific simulation.&quot;</span><span class="p">,</span> <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;simulations&quot;</span><span class="p">],</span> <span class="n">response_model</span><span class="o">=</span><span class="n">APISimulation</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">&quot;simulation-get&quot;</span><span class="p">,</span> <span class="p">)(</span><span class="n">views</span><span class="o">.</span><span class="n">simulation_get</span><span class="p">)</span> <span class="n">router</span><span class="o">.</span><span class="n">put</span><span class="p">(</span> <span class="s2">&quot;/simulation/</span><span class="si">{simulation_id}</span><span class="s2">/&quot;</span><span class="p">,</span> <span class="n">summary</span><span class="o">=</span><span class="s2">&quot;Update a simulation.&quot;</span><span class="p">,</span> <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;simulations&quot;</span><span class="p">],</span> <span class="n">response_model</span><span class="o">=</span><span class="n">APISimulation</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">&quot;simulation-put&quot;</span><span class="p">,</span> <span class="p">)(</span><span class="n">views</span><span class="o">.</span><span class="n">simulation_put</span><span class="p">)</span> <span class="n">router</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span> <span class="s2">&quot;/simulation/</span><span class="si">{simulation_id}</span><span class="s2">/&quot;</span><span class="p">,</span> <span class="n">summary</span><span class="o">=</span><span class="s2">&quot;Delete a simulation.&quot;</span><span class="p">,</span> <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;simulations&quot;</span><span class="p">],</span> <span class="n">response_model</span><span class="o">=</span><span class="n">APISimulation</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">&quot;simulation-delete&quot;</span><span class="p">,</span> <span class="p">)(</span><span class="n">views</span><span class="o">.</span><span class="n">simulation_delete</span><span class="p">)</span> </pre></div> <p>This should be pretty straightforward. <code>summary</code> and <code>tags</code> are used purely for the documentation site, they have no other functional purpose. <code>name</code> is used for getting a route&#8217;s URL by name, and <code>response_model</code> is used for validation of the response and documentation.</p> <p>Onto the views!</p> <h3>The views</h3> <p>The views are more straightforward than you&#8217;d expect. We have the GET/POST on the collection and GET/POST/PUT/DELETE on the object, but the heavy lifting is done by the <code>from_model</code>/<code>from_api</code> methods.</p> <p>Also, notice how authentication is done with the <code>Depends(get_user)</code> dependency, making it mandatory for each endpoint, and the <code>simulation</code> parameter is an actual <code>Simulation</code> model instance, not an ID. The model instance also gets validated to make sure it actually belongs to the user. We&#8217;ll see exactly how in a later section.</p> <p><strong><code>main/views.py</code></strong>:</p> <div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">Body</span> <span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">Depends</span> <span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">APISimulation</span> <span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">APISimulations</span> <span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">APIUser</span> <span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">DamnFastAPISimulation</span> <span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">Simulation</span> <span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">User</span> <span class="kn">from</span> <span class="nn">.utils</span> <span class="kn">import</span> <span class="n">get_simulation</span> <span class="kn">from</span> <span class="nn">.utils</span> <span class="kn">import</span> <span class="n">get_user</span> <span class="k">def</span> <span class="nf">simulations_get</span><span class="p">(</span><span class="n">user</span><span class="p">:</span> <span class="n">User</span> <span class="o">=</span> <span class="n">Depends</span><span class="p">(</span><span class="n">get_user</span><span class="p">))</span> <span class="o">-&gt;</span> <span class="n">APISimulations</span><span class="p">:</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Return a list of available simulations.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">simulations</span> <span class="o">=</span> <span class="n">user</span><span class="o">.</span><span class="n">simulation_set</span><span class="o">.</span><span class="n">all</span><span class="p">()</span> <span class="n">api_simulations</span> <span class="o">=</span> <span class="p">[</span><span class="n">APISimulation</span><span class="o">.</span><span class="n">from_model</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">simulations</span><span class="p">]</span> <span class="k">return</span> <span class="n">APISimulations</span><span class="p">(</span><span class="n">items</span><span class="o">=</span><span class="n">api_simulations</span><span class="p">)</span> <span class="k">def</span> <span class="nf">simulation_post</span><span class="p">(</span> <span class="n">simulation</span><span class="p">:</span> <span class="n">DamnFastAPISimulation</span><span class="p">,</span> <span class="n">user</span><span class="p">:</span> <span class="n">User</span> <span class="o">=</span> <span class="n">Depends</span><span class="p">(</span><span class="n">get_user</span><span class="p">)</span> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">APISimulation</span><span class="p">:</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Create a new simulation.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">s</span> <span class="o">=</span> <span class="n">Simulation</span><span class="o">.</span><span class="n">from_api</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">simulation</span><span class="p">)</span> <span class="n">s</span><span class="o">.</span><span class="n">save</span><span class="p">()</span> <span class="k">return</span> <span class="n">APISimulation</span><span class="o">.</span><span class="n">from_model</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">def</span> <span class="nf">simulation_get</span><span class="p">(</span> <span class="n">simulation</span><span class="p">:</span> <span class="n">Simulation</span> <span class="o">=</span> <span class="n">Depends</span><span class="p">(</span><span class="n">get_simulation</span><span class="p">),</span> <span class="n">user</span><span class="p">:</span> <span class="n">User</span> <span class="o">=</span> <span class="n">Depends</span><span class="p">(</span><span class="n">get_user</span><span class="p">)</span> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">APISimulation</span><span class="p">:</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Return a specific simulation object.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">return</span> <span class="n">APISimulation</span><span class="o">.</span><span class="n">from_model</span><span class="p">(</span><span class="n">simulation</span><span class="p">)</span> <span class="k">def</span> <span class="nf">simulation_put</span><span class="p">(</span> <span class="n">simulation</span><span class="p">:</span> <span class="n">Simulation</span> <span class="o">=</span> <span class="n">Depends</span><span class="p">(</span><span class="n">get_simulation</span><span class="p">),</span> <span class="n">sim_body</span><span class="p">:</span> <span class="n">DamnFastAPISimulation</span> <span class="o">=</span> <span class="n">Body</span><span class="p">(</span><span class="o">...</span><span class="p">),</span> <span class="n">user</span><span class="p">:</span> <span class="n">User</span> <span class="o">=</span> <span class="n">Depends</span><span class="p">(</span><span class="n">get_user</span><span class="p">),</span> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">APISimulation</span><span class="p">:</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Update a simulation.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">simulation</span><span class="o">.</span><span class="n">update_from_api</span><span class="p">(</span><span class="n">sim_body</span><span class="p">)</span> <span class="n">simulation</span><span class="o">.</span><span class="n">save</span><span class="p">()</span> <span class="k">return</span> <span class="n">APISimulation</span><span class="o">.</span><span class="n">from_model</span><span class="p">(</span><span class="n">simulation</span><span class="p">)</span> <span class="k">def</span> <span class="nf">simulation_delete</span><span class="p">(</span> <span class="n">simulation</span><span class="p">:</span> <span class="n">Simulation</span> <span class="o">=</span> <span class="n">Depends</span><span class="p">(</span><span class="n">get_simulation</span><span class="p">),</span> <span class="n">user</span><span class="p">:</span> <span class="n">User</span> <span class="o">=</span> <span class="n">Depends</span><span class="p">(</span><span class="n">get_user</span><span class="p">)</span> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">APISimulation</span><span class="p">:</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Delete a simulation.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">d</span> <span class="o">=</span> <span class="n">APISimulation</span><span class="o">.</span><span class="n">from_model</span><span class="p">(</span><span class="n">simulation</span><span class="p">)</span> <span class="n">simulation</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span> <span class="k">return</span> <span class="n">d</span> </pre></div> <h3>Utils</h3> <p>In this file we define methods to authenticate users and retrieve objects from the database by their ID while still making sure they belong to the authenticating user. <code>get_object</code> is a generic function to avoid repeating ourselves for every one of our models.</p> <p><strong><code>main/utils.py</code></strong>:</p> <div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Type</span> <span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">models</span> <span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">Header</span> <span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">HTTPException</span> <span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">Path</span> <span class="kn">from</span> <span class="nn">main.models</span> <span class="kn">import</span> <span class="n">Simulation</span> <span class="kn">from</span> <span class="nn">main.models</span> <span class="kn">import</span> <span class="n">User</span> <span class="c1"># This is to avoid typing it once every object.</span> <span class="n">API_KEY_HEADER</span> <span class="o">=</span> <span class="n">Header</span><span class="p">(</span><span class="o">...</span><span class="p">,</span> <span class="n">description</span><span class="o">=</span><span class="s2">&quot;The user&#39;s API key.&quot;</span><span class="p">)</span> <span class="k">def</span> <span class="nf">get_user</span><span class="p">(</span><span class="n">x_api_key</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">API_KEY_HEADER</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">User</span><span class="p">:</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Retrieve the user by the given API key.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">u</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">api_key</span><span class="o">=</span><span class="n">x_api_key</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">u</span><span class="p">:</span> <span class="k">raise</span> <span class="n">HTTPException</span><span class="p">(</span><span class="n">status_code</span><span class="o">=</span><span class="mi">400</span><span class="p">,</span> <span class="n">detail</span><span class="o">=</span><span class="s2">&quot;X-API-Key header invalid.&quot;</span><span class="p">)</span> <span class="k">return</span> <span class="n">u</span> <span class="k">def</span> <span class="nf">get_object</span><span class="p">(</span> <span class="n">model_class</span><span class="p">:</span> <span class="n">Type</span><span class="p">[</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">],</span> <span class="nb">id</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">x_api_key</span><span class="p">:</span> <span class="nb">str</span> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">:</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Retrieve an object for the given user by id.</span> <span class="sd"> This is a generic helper method that will retrieve any object by ID,</span> <span class="sd"> ensuring that the user owns it.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="n">user</span> <span class="o">=</span> <span class="n">get_user</span><span class="p">(</span><span class="n">x_api_key</span><span class="p">)</span> <span class="n">instance</span> <span class="o">=</span> <span class="n">model_class</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">user</span><span class="o">=</span><span class="n">user</span><span class="p">,</span> <span class="n">pk</span><span class="o">=</span><span class="nb">id</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">instance</span><span class="p">:</span> <span class="k">raise</span> <span class="n">HTTPException</span><span class="p">(</span><span class="n">status_code</span><span class="o">=</span><span class="mi">404</span><span class="p">,</span> <span class="n">detail</span><span class="o">=</span><span class="s2">&quot;Object not found.&quot;</span><span class="p">)</span> <span class="k">return</span> <span class="n">instance</span> <span class="k">def</span> <span class="nf">get_simulation</span><span class="p">(</span> <span class="n">simulation_id</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="o">...</span><span class="p">,</span> <span class="n">description</span><span class="o">=</span><span class="s2">&quot;The ID of the simulation.&quot;</span><span class="p">),</span> <span class="n">x_api_key</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">Header</span><span class="p">(</span><span class="o">...</span><span class="p">),</span> <span class="p">):</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> Retrieve the user&#39;s simulation from the given simulation ID.</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">return</span> <span class="n">get_object</span><span class="p">(</span><span class="n">Simulation</span><span class="p">,</span> <span class="n">simulation_id</span><span class="p">,</span> <span class="n">x_api_key</span><span class="p">)</span> </pre></div> <h3>Tests</h3> <p>The tests are very close to what you&#8217;re already used to from Django. We&#8217;ll be using Django&#8217;s testing harness/runner, but we can test FastAPI&#8217;s views by using FastAPI&#8217;s <code>client</code>. We&#8217;ll also be using Django&#8217;s/unittest&#8217;s <code>assert</code> functions, as I find them more convenient, but you can use anything you&#8217;d use with Django for those.</p> <p>As I mentioned earlier, the plain <code>TestCase</code> didn&#8217;t work for me, so I had to use the <code>TransactionTestCase</code>.</p> <p><strong><code>main/tests.py</code></strong>:</p> <div class="highlight"><pre><span></span><span class="c1"># We use this as TestCase doesn&#39;t work.</span> <span class="kn">from</span> <span class="nn">django.test</span> <span class="kn">import</span> <span class="n">TransactionTestCase</span> <span class="kn">from</span> <span class="nn">django.test.runner</span> <span class="kn">import</span> <span class="n">DiscoverRunner</span> <span class="kn">from</span> <span class="nn">fastapi.testclient</span> <span class="kn">import</span> <span class="n">TestClient</span> <span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">User</span> <span class="kn">from</span> <span class="nn">goatfish.wsgi</span> <span class="kn">import</span> <span class="n">app</span> <span class="c1"># A convenient helper for getting URL paths by name.</span> <span class="n">reverse</span> <span class="o">=</span> <span class="n">app</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">url_path_for</span> <span class="k">class</span> <span class="nc">TestRunner</span><span class="p">(</span><span class="n">DiscoverRunner</span><span class="p">):</span> <span class="k">def</span> <span class="nf">teardown_databases</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">old_config</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="c1"># This is necessary because either FastAPI/Starlette or Django&#39;s</span> <span class="c1"># ORM isn&#39;t cleaning up the connections after it&#39;s done with</span> <span class="c1"># them.</span> <span class="c1"># The query below kills all database connections before</span> <span class="c1"># dropping the database.</span> <span class="k">with</span> <span class="n">connection</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span> <span class="k">as</span> <span class="n">cursor</span><span class="p">:</span> <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span> <span class="sa">f</span><span class="s2">&quot;&quot;&quot;SELECT</span> <span class="s2"> pg_terminate_backend(pid) FROM pg_stat_activity WHERE</span> <span class="s2"> pid &lt;&gt; pg_backend_pid() AND</span> <span class="s2"> pg_stat_activity.datname =</span> <span class="s2"> &#39;</span><span class="si">{</span><span class="n">settings</span><span class="o">.</span><span class="n">DATABASES</span><span class="p">[</span><span class="s2">&quot;default&quot;</span><span class="p">][</span><span class="s2">&quot;NAME&quot;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#39;;&quot;&quot;&quot;</span> <span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Killed </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">cursor</span><span class="o">.</span><span class="n">fetchall</span><span class="p">())</span><span class="si">}</span><span class="s2"> stale connections.&quot;</span><span class="p">)</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">teardown_databases</span><span class="p">(</span><span class="n">old_config</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">class</span> <span class="nc">SmokeTests</span><span class="p">(</span><span class="n">TransactionTestCase</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="c1"># Warning: Naming this `self.client` leads Django to overwrite it</span> <span class="c1"># with its own test client.</span> <span class="bp">self</span><span class="o">.</span><span class="n">c</span> <span class="o">=</span> <span class="n">TestClient</span><span class="p">(</span><span class="n">app</span><span class="p">)</span> <span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="s2">&quot;user&quot;</span><span class="p">,</span> <span class="n">api_key</span><span class="o">=</span><span class="s2">&quot;mykey&quot;</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;X-API-Key&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">user</span><span class="o">.</span><span class="n">api_key</span><span class="p">}</span> <span class="k">def</span> <span class="nf">test_read_main</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">c</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">reverse</span><span class="p">(</span><span class="s2">&quot;simulations-get&quot;</span><span class="p">),</span> <span class="n">headers</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span> <span class="mi">200</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">(),</span> <span class="p">{</span><span class="s2">&quot;username&quot;</span><span class="p">:</span> <span class="s2">&quot;user&quot;</span><span class="p">})</span> </pre></div> <p>Keep in mind that, as the comment says, you need to use this custom runner, otherwise PostgreSQL connections don&#8217;t get cleaned up, for some reason. I&#8217;d love to figure out why, but after hours of debugging I didn&#8217;t manage to get anywhere, so I added this workaround.</p> <p>You need to add <code>TEST_RUNNER = "main.TestRunner"</code> to your <code>settings.py</code> to use that.</p> <p>That&#8217;s it!</p> <h2>Epilogue</h2> <p>I hope that was clear enough, there is a bit of confusion when trying to figure out which part is served by which library, but I&#8217;m confident that you&#8217;ll be able to figure it all out without much difficulty. Just keep in mind that FastAPI does everything view-specific and Django does everything else.</p> <p>I was legitimately surprised at how well the two libraries worked together, and how minimal the amounts of hackery involved were. I don&#8217;t really see myself using anything else for APIs in the future, as the convenience of using both libraries is hard to beat. I hope that asynchronicity will work when Django releases async support, which would complete this integration and give us the best of both worlds.</p> <p><a href="https://twitter.com/intent/user?screen_name=Stavros">Tweet</a> or <a href="https://mastodon.social/@stavros">toot</a> at me, or email me directly.</p> Mon, 11 May 2020 20:11:11 +0000 Make your own PCBs with a 3D printer https://www.stavros.io/posts/make-pcbs-at-home/ https://www.stavros.io/posts/make-pcbs-at-home/ <div class="pull-quote">More PCBs, less hassle</div><p>Listen, anyone can make a <span data-expounder="pcb">PCB</span> at home, it&#8217;s easy. <span data-expounded="pcb">PCBs (printed circuit boards) are those flat things with all the components that are inside all electronic devices, you&#8217;ve seen them.</span> All you need is a laser printer, some glossy magazine pages, print your circuit onto the page, use a clothes iron to transfer the toner onto your copper clad, if that doesn&#8217;t work use some water and some lacquer or something, I don&#8217;t know, I <a href="https://hackaday.com/2008/07/28/how-to-etch-a-single-sided-pcb/">stopped reading</a> at that point because the last time I saw a laser printer, a magazine and a clothes iron was in the nineties.</p> <p>Until recently, the only ways I knew to make PCBs was to practice the dark art above, to pay $10 and wait three weeks to get professional-looking PCBs from China, or to pay $60 and wait three days to get professional-looking PCBs from Europe. It was &#8220;cheap, fast, actually doable by a human person, choose two&#8221;.</p> <p>That always bugged me, it shouldn&#8217;t be like that, I have always been of the opinion that there shouldn&#8217;t be things you can&#8217;t make when you have a 3D printer, but PCBs have consistently eluded me. I yearned for them, I wanted to be able to make them at home, but it seemed impossible.</p> <p>One day, everything changed.</p> <!-- break --> <h2>That one day when everything changed</h2> <p>One day, I thought it would be fun to mount a pen onto my 3D printer and got it to draw <span data-expounder="stuff">stuff</span> <span data-expounded="stuff">(penises)</span> on paper. If you don&#8217;t know how a 3D printer works, now would be a good time to skim my <a href="https://www.stavros.io/posts/3d-printer-primer/">3D printer primer</a>, but in summary a 3D printer cuts an object into 2D slices and prints them on top of each other.</p> <p>If I could take just one of these slices and tell the printer to print higher, then the nozzle wouldn&#8217;t be touching the bed, I could mount the pen lower and put a piece of paper under it, and it would draw on the paper. Getting the printing software (called a slicer) to create a 3-dimensional shape out of a photo was easy, as you can just import a drawing and it will automatically convert it to a solid.</p> <p>I rubber-banded a Sharpie onto a holder I quickly designed, and, a few lines of code later, my printer produced this gem (excuse quality of potato):</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><video width="100%" controls><source src="plotter.mp4" type="video/mp4">Your browser does not support the video tag.</video></div><span class="caption">My 3D printer drawing my logo onto a piece of paper.</span></div><p>I felt so proud of the completely useless thing I had created, I immediately ran to share it <em>everywhere</em>. A friend saw it on Twitter and replied that this was <a href="https://twitter.com/_dzervas/status/1223509725486178304">a good way to make PCBs</a>.</p> <p>I was immediately intrigued.</p> <h2>Making PCBs with markers</h2> <p>Apparently, all that black magic about printing and transferring and magazines and lacquer above is just a way to keep the etching acid from dissolving the copper. You see, when you&#8217;re etching a PCB, you&#8217;re basically getting acid to dissolve the copper on your copper clad (the blank PCB). Except, you&#8217;re putting something (toner) over the copper on the parts you want to keep, so the acid can&#8217;t get to it, which leaves you with copper traces.</p> <p>It turns out you don&#8217;t really need to do all that stuff with the toner and printer, since you can cover the copper much more easily just by drawing over it with a permanent, waterproof marker (I used an Edding 140 S 0.3mm). I had inadvertently created a plotter, which can be used for painting a circuit onto a PCB.</p> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="pcb.png" data-lightbox="gallery"><img src="pcb-small.png"></a></div><span class="caption">This is a signal inverter for SmartPort telemetry.</span></div><p>I rushed to dig out the files for one of my old, small PCBs, which you can see on the right, and ran it through my improved drawing process. I had the printer draw it on paper, just to see what it would end up looking like, and the results were pretty encouraging.</p> <p>The paper somewhat absorbed the ink, as paper is wont to do, so there was some seepage, but overall the results looked like they should definitely work on copper. This is that first attempt:</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="paper-drawing.jpg" data-lightbox="gallery"><img src="paper-drawing-small.jpg"></a></div><span class="caption">That first attempt.</span></div><p>This looked very promising.</p> <h2>The improved drawing process</h2> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="copper.jpg" data-lightbox="gallery"><img src="copper-small.jpg"></a></div><span class="caption">The copper clad, placed on the printer.</span></div><p>The original process (where I exported stuff to an image and then converted the image to a 3D model with the slicer) worked well enough, but it was meant for working with 3D models, not circuits. As such, it could sometimes lose accuracy, which doesn&#8217;t really matter if you&#8217;re printing a figurine, but does matter if it means your circuit is no longer working.</p> <p>Fortunately, there is an excellent open source project called <a href="https://github.com/pcb2gcode/pcb2gcode/">pcb2gcode</a>, which converts <span data-expounder="gerber">Gerber files</span> <span data-expounded="gerber">(Gerber files are basically files that describe your PCB so the fabrication factory can make it)</span> to Gcode for various CAD tools to execute it. The Gcode that pcb2gcode outputs wasn&#8217;t compatible with my Marlin-style printer, so I adapted my script from above to turn the output to something my printer can run.</p> <p>My script does various things:</p> <ul> <li>It removes commands that are incompatible with or unsafe to run on my printer (e.g. drilling commands, as it has no drill, temperature and extrusion commands, as I don&#8217;t want it to heat up or extrude, etc).</li> <li>It rewrites comments so that they are compatible with my printer (pcb2gcode comments use parentheses, my printer uses semicolons).</li> <li>It makes sure all movements are within some specified coordinates, so the print head can&#8217;t go too low or too high by mistake.</li> <li>It allows me to offset all drawing commands by some distance, so I can print exactly where I want to.</li> </ul> <p>Unfortunately, as you can see on the photo of the paper drawing above, pcb2gcode has a bug where traces that are too narrow disappear, but the author is actively working on it and I worked around it in the mean time just by setting a lower marker tip diameter.</p> <p>Once everything was done on the programming side, and with lots of help from my friend <a href="https://andrewtop.com/">Andrew</a>, it was time to actually try and draw on copper. Here&#8217;s a video of the attempt:</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><video width="100%" controls><source src="drawing.mp4" type="video/mp4">Your browser does not support the video tag.</video></div><span class="caption">Drawing a PCB on copper clad.</span></div><div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="drawn.jpg" data-lightbox="gallery"><img src="drawn-small.jpg"></a></div><span class="caption">The drawn and cut design.</span></div><p>As the copper isn&#8217;t porous, the drawing looks much better than on paper. The traces are very accurately drawn, though the tolerances are a bit too tight for my liking. One trace at the top, especially, is way too close to the pad, though they aren&#8217;t actually touching. I&#8217;d have to see whether they would be bridged after etching, though.</p> <p>You also might notice a small hole in the very first pad, that&#8217;s because the ink hadn&#8217;t started running yet. It was easy to fix by just painting it in with the marker at the end.</p> <p>The only thing left to do is take some of that copper layer away so we&#8217;re left with our traces.</p> <h2>Etching</h2> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="scale.jpg" data-lightbox="gallery"><img src="scale-small.jpg"></a></div><span class="caption">Carefully weighing and diluting the sodium persulfate.</span></div><p>Surprisingly (and as my friend Josh rubbed in with what I imagine to be considerable glee), the actual etching part wasn&#8217;t as easy as I thought. It wasn&#8217;t <em>hard</em>, but it also didn&#8217;t consist of just throwing the PCB in acid and waiting.</p> <p>I&#8217;m still in a trial-and-error phase, as I&#8217;ve only etched two PCBs, but it seems like 100 grams of water with the appropriate amount of sodium persulfate dissolved in it is only good enough to etch a tiny PCB. If that&#8217;s true, it will take a lot of acid to etch a larger one, but I&#8217;ll have to experiment and see what happens.</p> <p><strong>UPDATE:</strong> Some commenters on HN pointed out that the reason this took so long was that I didn&#8217;t have a copper plane. I decided against that initially because I didn&#8217;t want the marker to have to basically paint the entire PCB, but after having etched, it&#8217;s definitely worth it. A plane makes it so that a lot of the copper is left on the PCB, and only the outlines of the traces are etched, which is all you need anyway. This way you aren&#8217;t wasting your etchant trying to etch a lot of copper, it takes less time, and your PCB ends up better etched.</p> <p>The other thing I&#8217;ve learned is that you shouldn&#8217;t heat the water above 50ish C, otherwise it starts to vapor and that can&#8217;t be good for anything.</p> <p>Also, the first time I etched a PCB I kind of just left it in the acid, but it turns out the copper dissolves much faster if you&#8217;re agitating or stirring the mixture. I started looking into designs for building an agitator for subsequent tries, but then realized that you don&#8217;t need one if you have a 3D printer, and wrote this Gcode file instead:</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><video width="100%" controls><source src="agitator.mp4" type="video/mp4">Your browser does not support the video tag.</video></div><span class="caption">You seem agitated.</span></div><p>It even heats the bed to 40 C so your etchant is kept warm! Is there anything 3D printers can&#8217;t do? Doesn&#8217;t look like it!</p> <h3>Etchant health concerns</h3> <p>I have no idea what the health concerns for these chemicals are. I&#8217;ve kind of just tried to avoid getting too close to the vat, even going as far as to put the lid on the tupperware container I was using, but I don&#8217;t know whether that delays or somehow impedes the etching process.</p> <p>It&#8217;s been hard to find info about all these chemicals, so if you know anything, please leave a comment below or send me an email, especially if you know of a better/more convenient/safer etchant. I will update the article with any info as it comes, for the edification of anyone who reads this.</p> <p><strong>UPDATES:</strong></p> <p><a href="https://news.ycombinator.com/item?id=22328419">throwanem on HN says</a> that sodium persulfate isn&#8217;t too bad, the biggest problem is fumes which can be kept to a minimum at low temperatures. Higher temperatures accelerate the etch, but the fumes shouldn&#8217;t be inhaled. It&#8217;s also a fire risk, and should be kept cool and dry.</p> <p>As always, any etchant should be safely disposed of and not poured down the drain.</p> <h2>The etching continues</h2> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="etching.jpg" data-lightbox="gallery"><img src="etching-small.jpg"></a></div><span class="caption">The PCB is bubbling.</span></div><p>Etching was otherwise uneventful, the PCB started bubbling at some point and then it started flaking, and the etchant turned slightly blue, which I assumed meant it was working. Agitating it a bit made it flake more, which seemed like a very good sign, so I just kept doing it and it kept working. I then left the mixture to its own devices because I was getting bored, until I noticed it was finally done.</p> <p>Eventually I was left with a slightly over-etched but otherwise extremely respectable PCB. I don&#8217;t know why the over-etching happened, I imagine it was because I left it in the etchant for half an hour. Maybe I need to use more etchant for it to go faster and not have time to over-etch, I hear it should ideally take from three to ten minutes.</p> <p>Regardless, with no small amount of pride, here&#8217;s my first ever home-etched PCB!:</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="etched.jpg" data-lightbox="gallery"><img src="etched-small.jpg"></a></div><span class="caption">The final product. I'm so happy.</span></div><p>It both looks good <em>and</em> works well! I tested everything with the multimeter and everything that should be connected is connected, and nothing that should be disconnected isn&#8217;t disconnected.</p> <p>All in all, I would call this an unconditional success!</p> <h2>This was all a ploy</h2> <p>Of course it was a ploy! What did you think, that I spent all this time, effort and tax payer dollars to create a signal inverter for an RC airplane? Don&#8217;t be naive.</p> <p>My ultimate goal was far more nefarious, and with Andrew&#8217;s help my life-long dream is now a reality. I present to you this:</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="dickbutt.jpg" data-lightbox="gallery"><img src="dickbutt-small.jpg"></a></div><span class="caption">The <a rel='nofollow' target='_blank' href='https://knowyourmeme.com/memes/dick-butt'>Richard Rear</a> PCB, the pinnacle of human innovation.</span></div><p>Yes, that is a dickbutt PCB with a red LED on the dick on its butt.</p> <p>If you don&#8217;t know whether to laugh or cry, you are exactly right, because that&#8217;s precisely what I was going for.</p> <h2>Epilogue</h2> <p>I am extremely excited about this process, as it finally allows me to easily make PCBs at home, quickly, conveniently, cheaply and at rather high tolerances, for what it is. I have been looking for something like this for ages, and I really think it could spark a minor revolution with hardware enthusiasts. It&#8217;s definitely much faster, better, more fun and more convenient than soldering protoboards, which I hate hate hate.</p> <p>If you want to do this at home, all my code, Gcode files and process documentation are in this repository:</p> <p><a href="https://gitlab.com/stavros/pcb-plotting">https://gitlab.com/stavros/pcb-plotting</a></p> <p>Feel free to open an issue if something isn&#8217;t working properly, or a PR if you want to improve something. I&#8217;d appreciate your help.</p> <p>Also, as always, please <a href="https://twitter.com/intent/user?screen_name=Stavros">Tweet</a> or <a href="https://mastodon.social/@stavros">toot</a> at me, or email me directly.</p> Fri, 14 Feb 2020 00:06:04 +0000 Behold: Ledonardo https://www.stavros.io/posts/behold-ledonardo/ https://www.stavros.io/posts/behold-ledonardo/ <div class="pull-quote">A revolutionary new invention that lets you take slightly different photos than before</div><p>A few years ago, I set out to reinvent photography. I didn&#8217;t have a good idea how to do this, I just knew I wanted to make something original, and combining photography with my electronics skills seemed like a good way to do that. It failed at reinventing photography, but I succeeded in writing a clickbait first sentence, and the process was lots of fun too.</p> <p>It all started one night, when I was having drinks with a friend and looking for something new to do with photography. Our conversation went something like this:</p> <ul> <li>I need to do something original with photography.</li> <li>Mmhmm.</li> <li>Maybe I could combine technology and photography to create something new, but what could it be?</li> <li>Hmm.</li> <li>I know! I&#8217;ll make a light stick thing.</li> <li>Mmm.</li> </ul> <p>The idea was great, I would use a LED strip to display images in mid-air, like those <a href="https://www.youtube.com/watch?v=4yuSBgBC35I">persistence of vision</a> displays. I could set the camera to record a long exposure, then move the strip and trace a pattern in the air. I had never seen anyone do this before, so the first thing I did was what everyone does when they have a groundbreaking idea: I searched the web to see if this already existed.</p> <!-- break --> <p>The second thing I did was what every self-respecting inventor does when they have a groundbreaking idea: I looked at the first two search results, saw that none of them resembled what I had in mind, said &#8220;Well, this conclusively proves that nothing like this has ever been done!&#8221; and started working on it.</p> <p>I call it&#8230; Ledonardo!</p> <h2>High-level idea</h2> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><img src="glamour-small.jpg"><a class="photocredit" href="https://learn.adafruit.com/adafruit-neopixel-uberguide/the-magic-of-neopixels" title="Photo from https://learn.adafruit.com/adafruit-neopixel-uberguide/the-magic-of-neopixels" target="_blank" rel="noopener noreferrer"><i class="fa fa-camera"></i></a></div><span class="caption">A strip of individually-addressable LEDs.</span></div><p>The basic idea is a device that will be a long strip of individually-addressable LEDs, which means that each LED on the strip can show a different color. I would also create a microcontroller with some software that allowed it to change the LEDs&#8217; colors in such a way that they could show photos and patterns in the air. Since a long bar is almost one-dimensional, to show photos and patterns you&#8217;d have to move the bar along the path you want the image to show, while the camera&#8217;s shutter remains open. This means you can pretty much only use this for night photography or light painting, but the result would be fairly impressive.</p> <p>Because it&#8217;s hard to explain, especially to people with no experience in long-exposure photography, here&#8217;s a video of how taking a photo with the final version of the bar (spoiler alert) works:</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><video width="100%" controls><source src="ledonardo.mp4" type="video/mp4">Your browser does not support the video tag.</video></div></div><p>Basically, you keep the camera shutter open for many seconds, move a light source in front of the camera, and (if you&#8217;ve done everything right) the trajectory of the light source shows up on the photo.</p> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><img src="wemos-small.jpg"></div><span class="caption">The WeMos D1 mini.</span></div><p>I already had some WS2812 LED strips and I decided to use the ESP8266, my go-to microcontroller. The ESP8266&#8217;s built-in WiFi is extremely handy for cases where you have complex interfaces, since you can use a mobile phone to control it. That way, I could upload photos from the phone to display on the strip.</p> <p>Please excuse the draft quality of the photos in the rest of the article, they are quick tests I made while making the bar.</p> <h2>Hardware details</h2> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="controlbox.jpg" data-lightbox="gallery"><img src="controlbox-small.jpg"></a></div><span class="caption">The control box, 3D-printed case and battery pack.</span></div><p>The hardware didn&#8217;t really change at all from the first iteration. I&#8217;m using a WeMos D1 mini as the ESP8266 board, with its 5V pin connected to two 18650 3.7V batteries in series, which usually provides power for around two hours of shooting, depending on LED intensity. The LEDs are also connected to the batteries and the output pin of the WeMos, and that&#8217;s pretty much it for the schematic. I also added a small switch so I can cut power to the whole assembly when I&#8217;m not using it.</p> <p>The other hardware-related aspect of this is a small box I designed and 3D printed to house the microcontrollers. I did this because otherwise the microcontroller would dangle from the wires and they&#8217;d frequently get cut. With the box, everything fits neatly inside, all the wires are held by the edges of the box so the stress concentrates on the plastic case and wire shielding, and there is no tension on the solder joints.</p> <h2>First attempt, network-based, sparse strip, square pixels</h2> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="bar.jpg" data-lightbox="gallery"><img src="bar-small.jpg"></a></div><span class="caption">The LED strip glued onto a wooden bar.</span></div><p>I ran some back-of-the-envelope calculations with the image dimensions, bits per pixel, etc and realized that the 2 MB storage of the ESP8266 would only be enough to fit one or two images. I would also need to recompile and deploy the firmware every time I changed something, which I didn&#8217;t want to do. Instead, I decided to send the image over the WiFi connection and simply display it on the strip. This had the added advantage that the computer handled the complicated parts, i.e. resizing the image, timing it according to the speed you walked, etc. All the microcontroller had to do was display each column as it received it.</p> <p>Another consideration to pay attention to was image dimensions. Since the strip only had 50 LEDs (30 per meter), a square image would be resized to 50x50 pixels, which was pretty small but hopefully usable. LED strips with double the LED density exist, and I ordered one, but it would take a month before it arrived so I would have to make do with the sparse one for now.</p> <p>You can see the results of this initial version in the images below:</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="firstface.jpg" data-lightbox="gallery"><img src="firstface-small.jpg"></a></div><span class="caption">A photorealistic portrait, to test color reproduction and resolution. You can see the low resolution and bad color reproduction, but it works great for a first try.</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="firstlogo.jpg" data-lightbox="gallery"><img src="firstlogo-small.jpg"></a></div><span class="caption">A circular logo with a white symbol in the middle. The circle has stair-step edges and the symbol is almost invisible.</span></div><h2>Second attempt, network-based, dense strip, non-square pixels</h2> <p>My new strip finally arrived, with twice the number of LEDs per meter! This meant a higher vertical resolution, which means sharper images!</p> <p>While testing with higher-resolution images, I realized that, even though I only had 100 vertical pixels to play with, that didn&#8217;t mean I had to constrain myself to 100 horizontal pixels as well! Since the horizontal resolution was only governed by colors changing at various times instead of how far apart LEDs were spaced, I could theoretically get infinite horizontal resolution. Practically, I&#8217;m still limited by how fast the LEDs can change colors, but that&#8217;s the only limitation. If my LEDs can change colors every 10 ms, I can get 100 pixels per second, which is around a pixel per cm at walking speed.</p> <p>Unfortunately, going over the network was problematic, as I only managed to get 5 pixel changes per second in my tests. That was much better than before, but still not great.</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="secondface.jpg" data-lightbox="gallery"><img src="secondface-small.jpg"></a></div><span class="caption">Another bad shot of the portrait, but you can see that the horizontal resolution is greatly improved.</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="secondlogo.jpg" data-lightbox="gallery"><img src="secondlogo-small.jpg"></a></div><span class="caption">The circle actually looks circular now!</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="batman.jpg" data-lightbox="gallery"><img src="batman-small.jpg"></a></div><span class="caption">The new, dense LED strip allows for better vertical resolution as well, and images come out great as long as you specify your walking speed in the app (which I didn't do perfectly here).</span></div><h2>More testing</h2> <p>Another problem with going over the network is that sometimes you get <span data-expounder="dropped">dropped packets</span>. <span data-expounded="dropped">This means that, when the computer sends the column, it is lost in transmission for whatever reason, and the microcontroller never receives it, so it can&#8217;t display it.</span> If you look closely at the images, you can see that sometimes a column will not have arrived, which leads the firmware to just repeat the previous column. This creates jagged edges and repeated columns in detailed images.</p> <p>The images below illustrate this, where the white in the eye of the portrait is square and the circular logo has lots of jagged edges.</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="beachface.jpg" data-lightbox="gallery"><img src="beachface-small.jpg"></a></div><span class="caption">Notice the eye on our right, where most detail is lost.</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="glitchylogo.jpg" data-lightbox="gallery"><img src="glitchylogo-small.jpg"></a></div><span class="caption">The circle is not circular, there are jagged edges on the left and right where packets were dropped.</span></div><h2>Python rewrite, SD card, color problem</h2> <p>Due to the network issues detailed above, I decided to rewrite the whole thing using an SD card. I also wanted to write a converter program to take any image type (JPEG, PNG, etc) and convert it to a bitmap that could be easily and quickly read from the card and displayed. This would improve display, because there would be no more dropped packets. It would also greatly increase horizontal resolution, as reading from the SD card is much faster than going over the network, so you get more color changes per second.</p> <p>I decided to write the microcontroller firmware in microPython this time, because I had really started to hate C. The tooling that resized the images and generated the bitmaps on the computer side was also written in Python.</p> <p>I got a simple SD card shield for the WeMos, which microPython made trivial to use (you basically just read the file system as normal). It only took a day to rewrite everything, as much of the code of the image-sending program was already in Python. Unfortunately, and much to my dismay, reading the pixels and writing them one by one to the LEDs was taking so long that I could only display 8 columns per second. I spent quite some time trying to optimize this, but there&#8217;s only so much optimization you can do to a two-line <code>for</code> loop.</p> <p>Jumping into the micropython source for the LED library implementation, I realized I didn&#8217;t even have to keep the <code>for</code> loop! I could just directly assign the data I read from the SD card to the internal buffer of the LED library instance and display it. This sped my code up tenfold, and now I could cram much more detail in a centimeter of display!</p> <p>However, I now had a different problem! The colors were off, as you can see in the images below.</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="badmonalisa.jpg" data-lightbox="gallery"><img src="badmonalisa-small.jpg"></a></div><span class="caption">The Mona Lisa, looking green.</span></div><h2>Color problem fixed</h2> <p>Apparently, someone decided to make the WS2818 LED addressing order &#8220;green, red, blue&#8221; instead of &#8220;red, green, blue&#8221; that is the standard. That took a bit of debugging, but I eventually found the problem and quickly fixed it. Now all the colors showed up as they should!</p> <p>I still didn&#8217;t like the striped LED lines that appeared because the LEDs were basically points of light, I wanted the final result to look a lot more &#8220;solid&#8221;. However, I did figure out that the LEDs were <em>way</em> too bright for the camera&#8217;s settings, which caused the images to appear washed out. I turned the brightness down to 1% of the total brightness, which made the images look <em>much</em> nicer!</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="goodmonalisa.jpg" data-lightbox="gallery"><img src="goodmonalisa-small.jpg"></a></div><span class="caption">A much less green-looking Mona Lisa.</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="goodtree.jpg" data-lightbox="gallery"><img src="goodtree-small.jpg"></a></div><span class="caption">The test tree image (shown here for the first time) was also looking much better.</span></div><h2>Diffuser, gamma correction!</h2> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><img src="u-channel-small.jpg"></div><span class="caption">A plastic U-channel.</span></div><p>I began thinking about how I could create a light diffuser, to get the images looking much more even and less like a zebra. To do that, I would need to use some sort of semi-opaque and color-neutral material, such as paper or, even better, tracing paper. However, I also needed to put some space between the diffuser and the LED, to allow the light some space to actually diffuse.</p> <p>I considered designing and 3D-printing spacers that would clip onto the wooden bar and wrapping pieces of paper around them, but that was a bit too difficult to make and damage-prone. I visited my local hardware store and discovered plastic U-channels for running wires through. These are basically a long, square plastic pipe with one side exposed, which was perfect for me.</p> <p>I pulled on the sides of the channel a bit to open up, so I could get a wider angle instead of 90 degrees, which would in turn give me a wider bar. This worked pretty well, although the whole thing looks like the hacked-together home prototype it is.</p> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="channelbar.jpg" data-lightbox="gallery"><img src="channelbar-small.jpg"></a></div><span class="caption">A side view of the bar, lightstrip in U-channel and masking tape diffuser on top.</span></div><p>I glued the LED strip along the bottom side of the U-channel, soldered the wires that ran to the microcontroller on the exposed end and connected everything up. I then ran some masking tape over the open end of the channel and stuck it to its two sides.</p> <p>I also came across a vital piece of information somewhere, completely at random: LEDs don&#8217;t have a linear response! To represent a color as our eyes see it, you can&#8217;t just set the RGB led to those color&#8217;s values, you first need to perform what is called <a href="https://en.wikipedia.org/wiki/Gamma_correction">Gamma correction</a> so that the colors will show up correctly. Fortunately, this was very easy to do in the converter program, so I quickly implemented that as well.</p> <p>I tried some test photos, and the results looked <em>amazing</em>. You still need a steady hand, but the LEDs themselves work great.</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="monalisa.jpg" data-lightbox="gallery"><img src="monalisa-small.jpg"></a></div><span class="caption">The Mona Lisa, looking even better than the orignal. I'm sorry, Leonardo.</span></div><div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="maria.jpg" data-lightbox="gallery"><img src="maria-small.jpg"></a></div><span class="caption">Abstract patterns with a human model (my friend <a href='https://www.instagram.com/ksd_maria/'>Maria</a>).</span></div><h2>Done</h2> <p>At this point, I consider the project done. It&#8217;s good enough to use, very sturdy, and works great, but one thing surprised me negatively. I thought that it would be fantastic to be able to display photorealistic images in mid-air, but it turned out that abstract patterns and colors make for <em>far</em> more interesting photographs. If I knew this from the start, I might not have spent so much time obsessing about the horizontal resolution, since it doesn&#8217;t really matter.</p> <p>Another thing I learnt long after I started working on this is that someone else had, indeed, made this. They called theirs <a href="http://www.thepixelstick.com/">the pixel stick</a>, whereas I don&#8217;t even know what to call mine yet. I think they beat me to it only slightly, but it doesn&#8217;t really matter, since I did this for fun and I had a lot of that.</p> <p>As a parting gift, here are some more &#8220;production&#8221; images that I shot with Ledonardo:</p> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="alexandra.jpg" data-lightbox="gallery"><img src="alexandra-small.jpg"></a></div><span class="caption">Projecting an image of wings behind the model. Unfortunately, there's a wall behind her, so the wings look projected there and ruin the effect a bit.</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="tree.jpg" data-lightbox="gallery"><img src="tree-small.jpg"></a></div><span class="caption">A tree image.</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="wings.jpg" data-lightbox="gallery"><img src="wings-small.jpg"></a></div><span class="caption">Abstract pattern behind a model, the shape is done by moving the bar in a wing pattern behind the model.</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="lastface.jpg" data-lightbox="gallery"><img src="lastface-small.jpg"></a></div><span class="caption">The test portrait, you can see that color reproduction is excellent.</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="stairs.jpg" data-lightbox="gallery"><img src="stairs-small.jpg"></a></div><span class="caption">Some abstract patterns on a stairway.</span></div> <div class="clearfix"></div><div class="aligncenter"><div class="photo-container"><a href="swirls.jpg" data-lightbox="gallery"><img src="swirls-small.jpg"></a></div><span class="caption">A "proper" photo with an abstract pattern throughout the room.</span></div><p>That&#8217;s all for this project, I hope you enjoyed reading about it as much as I enjoyed making it! I&#8217;ve uploaded the code to a git repo, so if you want to take a look at it just visit <a href="https://gitlab.com/stavros/ledonardo">the ledonardo repository</a>.</p> <p>If you want to see more of my photography-related escapades, I post my photos on Instagram under the handle <a href="https://www.instagram.com/stavroskorok/">stavroskorok</a> and my maker-related things on IG at <a href="https://www.instagram.com/stavrosware/">stavrosware</a>. You can also follow me on <a href="https://twitter.com/intent/user?screen_name=Stavros">Twitter</a> or <a href="https://mastodon.social/@stavros">Mastodon</a>.</p> <p>I&#8217;d also like to thank Maria, Alexandra and Anna for their help and support.</p> <p>If you have any feedback or questions, please leave a comment below!</p> Tue, 01 Oct 2019 21:23:59 +0000 Seven tips for a great remote culture https://www.stavros.io/posts/seven-tips-great-remote-culture/ https://www.stavros.io/posts/seven-tips-great-remote-culture/ <div class="pull-quote">Make Remote Great Again</div><p>Did you like that clickbait title? I&#8217;ve been practicing. This article doesn&#8217;t contain seven tips because I hate listicles. It&#8217;s just a recounting of my experience working remote for fifteen years now and observations on what works and what doesn&#8217;t, but it doesn&#8217;t matter, because the amazing title has piqued your interest.</p> <p>For a bit of background, my first job was working in an office, as IT support for a construction company. I did that for three years, and then I got a remote job and never looked back. Personally, I enjoy the freedom that comes with being able to work from anywhere, and I&#8217;m lucky enough to be one of the people who can. Many of my friends have to be at their home office or a coworking space to get work done, but I can focus anywhere, which allows me to travel to another country for a week or two and work from there.</p> <p>I&#8217;m not going to go into the pros and cons of remote working, I assume they&#8217;ve been beaten into you by the myriad of other posts, since it&#8217;s a trendy topic. Instead, I&#8217;ll assume you are interested in improving your existing remote culture and I&#8217;ll detail <!-- break --> what has worked well for me.</p> <h2>Improving communication</h2> <p>From 2012 to 2019 I worked at Silent Circle, where we (rather inadvertently) created one of the best remote cultures I&#8217;ve worked in. The company had a central office, but engineering was entirely remote, which worked very well. In my experience, having some engineers be on-site and some remote never works (that&#8217;s right, I said never, I&#8217;ll never become a pundit if I equivocate). What usually tends to happen is that the on-site people communicate (and gossip) with each other much more easily, so the communication channels remote people have access to become second-tier.</p> <p>My impressions for what works are based on the way things worked at Silent Circle, so that&#8217;s what I&#8217;m going to detail. If you have any feedback or if other things worked for you, please leave a comment below.</p> <p>The biggest differentiators I can identify between that culture and others are three:</p> <ol> <li>IM was almost exclusively one-to-one.</li> <li>Email was used almost exclusively for any sort of decision or any discussion that required more than two people.</li> <li>We had physical desk phones that we configured to very high audio quality.</li> </ol> <h3>One-on-one IMs</h3> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="remote-position.jpg" data-lightbox="gallery"><img src="remote-position-small.jpg"></a><a class="photocredit" href="https://unsplash.com/photos/GpXII6QrPw0" title="Photo from https://unsplash.com/photos/GpXII6QrPw0" target="_blank" rel="noopener noreferrer"><i class="fa fa-camera"></i></a></div><span class="caption">Only too late did Jeremy realize what the job posting meant when it said "remote position".</span></div><p>One-on-one IMs were fantastic for socializing. We used a self-hosted Jabber server with Pidgin/Adium/whatever IM client each of us preferred. We did have some group chat rooms, but they were almost never used, in stark contrast to companies that use Slack, where most communication happens in public channels. This made socializing much easier, as it&#8217;s harder to socialize in a room where everyone can hear you, which led to everyone getting to know each other better.</p> <p>When you needed something from a coworker you didn&#8217;t know well, you would usually IM them and you&#8217;d talk about the business need. Once that was done, you were much more free to talk about your personal life, chat about some topics or just be more informal and relaxed. This isn&#8217;t impossible in chat rooms, but you&#8217;re much more guarded when you know that the entire company might be watching.</p> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="standup.jpg" data-lightbox="gallery"><img src="standup-small.jpg"></a><a class="photocredit" href="https://unsplash.com/photos/10Zxsivo4LY" title="Photo from https://unsplash.com/photos/10Zxsivo4LY" target="_blank" rel="noopener noreferrer"><i class="fa fa-camera"></i></a></div><span class="caption">On-site team members <strong>must not</strong> be allowed to talk to each other. Buy them headphones.</span></div><p>Since we&#8217;d chat to each other privately in the course of work, we quickly became friends with each other and would chat to each other outside business contexts too. This made it much more likely to discover what was going on in what they were working on too. For example, my friend who worked on support would chat about the increased workload lately, I&#8217;d ask why that was and he&#8217;d say that people had problems with feature X. Usually, this was an easy fix for me, but support didn&#8217;t know enough to ask for the fix, they just thought that&#8217;s the way things were.</p> <p>The increased socialization gave people opportunities to help each other, it increased how well people worked together and broke down silos. In my opinion, these behaviours improved the company&#8217;s products and kept morale higher, as well.</p> <p>An important topic that might go without saying is that we all flew to the head office to meet in person once or twice a year. The on-site meet is crucial for getting to know the people you don&#8217;t normally get a chance to talk to (e.g. people you don&#8217;t usually work with). It also helps team members to bond and get to know each other at a more personal level.</p> <h3>Using email for discussions</h3> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="mechanical-keyboard.jpg" data-lightbox="gallery"><img src="mechanical-keyboard-small.jpg"></a><a class="photocredit" href="https://unsplash.com/photos/nCvi-gS5r88" title="Photo from https://unsplash.com/photos/nCvi-gS5r88" target="_blank" rel="noopener noreferrer"><i class="fa fa-camera"></i></a></div><span class="caption">Unfortunately, their choice of loud mechanical keyboards usually makes remote workers social pariahs.</span></div><p>Any discussion that required more than one person would take place over email, with the exception of tickets/PR discussions that happened on our bug tracker (Gitlab). The ticket communication flow was very similar to email, since the two forms are pretty similar, and I will treat them as interchangeable here.</p> <p>The benefits of using email for discussions are pretty self-evident: Having the luxury of time while when expressing your opinion encourages you to think longer about what you&#8217;re saying, flesh your argument out more and come up with better arguments on a topic. Slack is roughly on par with face-to-face chats in that regard, it doesn&#8217;t really give you enough time to build your ideas. Being able to quote parts of another person&#8217;s argument when replying to them while building a larger narrative to argue your point makes discourse more fruitful.</p> <p>This makes email more asynchronous as well, so team members in different timezones are still part of the discussion and don&#8217;t feel like they&#8217;ve missed a bunch of points that have scrolled past. I would routinely wake up, browse the conversations that west-coast people were having the previous night, gather up all their arguments and reply to them one by one with my observations. If this took place on Slack, it would be a bunch of disjoint points that would have scrolled past long ago, while the discussion moved on.</p> <h3>Desk phones</h3> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="remote-worker.jpg" data-lightbox="gallery"><img src="remote-worker-small.jpg"></a><a class="photocredit" href="https://unsplash.com/photos/WHNkkoHDUJY" title="Photo from https://unsplash.com/photos/WHNkkoHDUJY" target="_blank" rel="noopener noreferrer"><i class="fa fa-camera"></i></a></div><span class="caption">The average remote worker, hard at work.</span></div><p>I love love <em>love</em> desk phones. We had Polycom phones whose hardware was excellent, the microphone was outstanding and usability was just top-notch. We did have a team of people who tuned them expertly, which helped make the sound quality the best I&#8217;ve ever heard on a phone, but there exist companies that will run and tune your PBX for a fee.</p> <p>You could press a single button and immediately call anyone in the company, and the convenience of having a high-quality speakerphone was unparalleled. It was as if the person was right there in the room with you, so much so that we&#8217;d sometimes just call each other and leave the line open while we worked so we could feel like we were working next to each other. I can&#8217;t understate how popular the phones were and how beneficial an impact they had. Their main use was as a replacement for meeting face to face, and they performed admirably.</p> <p>Nowadays I use Zoom, but the difference in usability is night and day. Having physical buttons for mute, calling, being able to pick up and hang up calls, and fantastic handsfree quality made all the difference. I really miss the phones every day, and I think they&#8217;re an expense that pays for itself many times over.</p> <h2>Defaults matter</h2> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="off-button.jpg" data-lightbox="gallery"><img src="off-button-small.jpg"></a><a class="photocredit" href="https://unsplash.com/photos/cw_uvISXkCI" title="Photo from https://unsplash.com/photos/cw_uvISXkCI" target="_blank" rel="noopener noreferrer"><i class="fa fa-camera"></i></a></div><span class="caption">Fun fact: Apple put the "off" button out of the way so that no one would ever think to press it.</span></div><p>Nothing we did on 1:1 chats is impossible to do in Slack-style group chats (or various other setups). Conversely, you can have a conference call or a chat on Zoom, but the experience is significantly worse than with a physical phone. Defaults matter, and that&#8217;s basically what culture is: A collection of defaults and slight encouragements that shape people&#8217;s behavior by gently nudging it in certain directions.</p> <p>Just like <a href="http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html">slightly slower web pages lead to massive dropoffs in traffic</a>, slight increases in ease of communication lead to massive improvements in communication quality (<a href="https://en.wikipedia.org/wiki/Source">source</a>). The suggestions in this article might seem like small improvements, but they add up, and they help business-related discourse as well as the equally-important socializing.</p> <h2>What I currently prefer</h2> <p>Since I left Silent Circle, I&#8217;ve mainly worked with companies that used <span data-expounder="slack">Slack</span> for communication, which I very much dislike. <span data-expounded="slack">I&#8217;m using &#8220;Slack&#8221; here as a catch-all term for all software that&#8217;s similar, like HipChat, etc, not as a criticism of Slack specifically.</span> It flies in the face of everything I mentioned above, and promotes quick, minimal-thought replies. It is possible to use Slack to socialize, and I do it too, but that&#8217;s in spite of what Slack promotes, not because of it.</p> <p>I have since come across <a href="https://zulipchat.com/">Zulip</a> and I like it a whole lot (this post is not a plug for Zulip I swear, I just honestly find it excellent). By default, it encourages you to write long-form messages with rich text editing (for example, enter changes lines by default). Every new discussion happens in a new topic, which is very reminiscent of email, so it&#8217;s basically a cross between group chats and email.</p> <div class="clearfix"></div><div class="alignright"><div class="photo-container"><a href="quran.jpg" data-lightbox="gallery"><img src="quran-small.jpg"></a><a class="photocredit" href="https://unsplash.com/photos/r8H8K3w9AzA" title="Photo from https://unsplash.com/photos/r8H8K3w9AzA" target="_blank" rel="noopener noreferrer"><i class="fa fa-camera"></i></a></div><span class="caption">Kristen's emails were always long and well-thought-out. Unfortunately, they were in Arabic, which nobody spoke.</span></div><p>It allows you to have long, tidy discussions that never move past the topic, allowing you to contribute later and with more thought-out replies. Additionally, I like establishing a guideline where private chats can be as quick/informal as you like, but public chats should be longer-form and higher thought (basically like composing an email). Also, Zulip&#8217;s keyboard accessibility is stellar, and you can easily navigate among discussions and topics with a few keystrokes.</p> <p>Unfortunately, I haven&#8217;t found a replacement for physical phones, which I still consider indispensable. I can wholeheartedly recommend investing in good phones and a PBX service, on which I&#8217;m sure the returns will be massive.</p> <h2>Epilogue</h2> <p>This is my experience with what helps a remote culture grow. I know that many people have different views on this, and I&#8217;d be very interested in hearing about yours, if you&#8217;ve worked with a different setup that worked very well for you. I&#8217;m especially curious to hear about setups that brought you closer as a team and helped you move from &#8220;coworkers&#8221; to &#8220;friends&#8221; like IM did for us.</p> <p>Please leave a comment below! Also, you can <a href="https://twitter.com/intent/user?screen_name=Stavros">Tweet</a> or <a href="https://mastodon.social/@stavros">toot</a> at me, or email me directly.</p> Mon, 29 Jul 2019 16:45:38 +0000