A Jumped-Up Pantry Boy

Wow, I have been working on this project for a long time. This has been a side project through-and-through: a pursuit of passion that has lived in the margins of jobs and urgent political projects. The above graphs are commits on the two biggest components of swhenever.com: the backend API and the React frontend app. You can see periods of intense activity and inactivity; some focus alternating between frontend and backend; and how much more domain logic is in the backend.

I proudly and officially released version 1.0 at the beginning of January. It’s a good time for reflections.

Domain-Driven Design worked well

Not pictured in the commit graphs: lots of hours of hashing out the domain back in 2015. I had read a lot of Implementing Domain-Driven Design and found the ideas compelling. This project was my first attempt to apply them seriously, and the team at the time (primarily Jim Cummins, Jeremy Ward and me) brainstormed and disagreed and documented domain concepts before doing serious coding. (I had a brittle proof-of-concept before that).

I think it was worth it. The resulting domain concepts (“ubiquitous language”) feature prominently in the code and have been really helpful in guiding architecture. I still think about the system in those terms even though I haven’t referenced/modified the planning documents for a couple years.

Take MVP a little more seriously

Honestly, there are parts of the system that are pretty over-engineered. I’ve cut back on aspects of it, but could go farther. Used CouchDB to store JSON documents that should have just been a longtext field in MySQL. Used RabbitMQ to serve up a couple job queues that should have just used laravel’s MySQL queue provider. It was worth the time to think through the domain and code to those abstractions, even though it was probably slower. But it was not worth the time to learn pieces of technology that will theoretically provide benefits later. Neither would have been expensive to switch to later on. That said …

I have learned so, so much

Taking up this project and sticking with it - regardless of serious stretches on the back burner - has been the best technology learning experience of my life. If I ever want to justify the slow burn or the overengineering to myself, this is it. I’ve learned Python, Docker, React, and Kubernetes. I’ve tackled the challenges of designing, building and deploying a complex system from alpha to omega. Almost three years on, sitting down to work on this is still the most rewarding and pleasurable work I’ve ever done in technology.

It’s just been an incredible blessing to make the space for a project that I deeply believe in. I’m able to do that because of a lot of privilege: specifically because I’ve worked part-time while still pulling a salary around the US household median income. Everybody deserves a life of dignity and creativity, but instead of producing that, our political economy has given all the productivity gains for 40 years to the 1%. I think about the local artists I’ve known or followed over the years and it staggers me how much more beauty the world would have in it if those folks just had a stable economic position. Artists are some of the hardest working people I know, who take some of the most risk in service of an ideal. If swhenever.com gets a little more recognition to these folks I will be very satisfied.

Slow and steady works OK

I wish I would have gotten this out to a broader audience earlier. Software that nobody is using, by definition, isn’t terribly useful. The intrinsic joys of the project are seductive: I honestly could probably have never launched anything and still really enjoy building it. But on the plus side, the system is working pretty well. It’s been working for a while. I really wanted the project to be sustainable, with as little manual data entry as possible.

It’ll be interesting if people engage with the recommendation feature. That’s the part that probably has the least basis in measureable user feedback (as opposed to my own hopes/vision), and I delayed launch to build that out. If it doesn’t get much engagement, then that would be a pretty clear example of indulging too much waterfall-style software development.

But hey, First Avenue just DM’d me, so it’s all good :)

This recording from a different show doesn’t do it justice, but holy cow it was such a joyous surprise last night when the ‘Snakes broke out a raucous version of one of my favorite Dylan shows.

Hook & Ladder is a nice little hall! I miss Patrick’s rainbow flag atop the old firehouse, but I dig it as a music venue.

I had the opportunity to catch several sessions on conversational interfaces (you know, BOTS) at re:invent and wanted to collect some notes that I found interesting.

One point was: don’t launch early with a convo interface that is just kind of barely working. The data you’ll get once deployed will actually be skewed by the deficiencies in your interface/bot. Instead launch a greater competence w less data, then your real data will have more meaning and drive better performance

Rubric for thinking about success:

• Smart
  • What can you do
  • How useful is it?
  • Am I training you or are you trainings me to use you?
• Trustworthy
  • How dependable?
  • How performamt?
  • How well do you know me?
• Delegate
  • I trust you
  • I can take some services for granted
  • I can focus on other decisions

Advanced Dialog attributes

• Subdialog
  • Repeat
  • Refime
  • Correct
  • Confirm,
• Mixed initiative
  • User may answer in a way that doesn’t address question, but instead expresses a desire to focus on something else
  • Be good at things related to what you’re asking user about
  • Adding constraints when replying to a req for input
• Exception handling
  • Handle exceptions to rules
• Speech acts
• Interpretation in context: partial info makes sense in context
  • Anaphora
  • World state
  • Features
• Personalization
• Proactivity
• Complex language
• Meta dialogue, talk about the dialogue itself, self awareness

State of the art is not available in a deployed toolkit. Have to search for what tools libraries to put together.

AWS Lex looks like a pretty easy way to create a conversational bot (text or audio). Very little programming required. It seems very goal oriented (define intents and inputs - slots - needed to fulfill those intents) rather than requiring you to design a big branching conversation hierarchy. Got some good notes on an architecture to support doing more sophisticated things with Lex. Limitations from the speaker: limited ability to correct Lex’s interpretations of users, cannot easily set a slot/input to be conditionally required (eg based on user session), need to route user to sub-bots in order to handle more complex/branching conversations.

If you’re not yet convinced that DevOps changes lives, this is the video to watch. Presented by the researchers behind the latest State of DevOps report. Very good practical advice about the roles of cultural/leadership changes alongside changes in technique/practice.

Research found which strategies meaningfully correlated with high IT performers, and demonstrates that high IT performance is likewise strongly correlated with higher performance on overall enterprise goals. Lots of good examples and dissection in all three areas.

One interesting claim was that they found no tradeoffs in enterprises that implemented lots of tools/processes oriented towards quality of software/deployment. Enterprises did NOT experience more failures as a result of deploying more changes. High performers had less time to release changes, higher release frequency, lower time to restore service, and lower change fail rate.

Lots of interesting points about organizational culture: pathological vs. bureaucratic vs. generative (see PDF they referenced)

I previously wrote about Hyper.sh as a nice step-up from Docker Compose: a good way to deploy an environment that is working as a Compose file without learning new orchestration tech. That still holds true, but after the experience of trying to make my deployment more robust on Hyper.sh, I’ve arrived at an important caveat: unless you’re in production.

Hyper.sh was great for experimentation and getting a cluster of containers up quickly. However, I ended up experiencing a number of instances where the service behaved inconsistently. I ran into more and more cases where it seemed like their basic building blocks were missing the features and glue to smoothly roll out a complex solution.

So my message to past-me is: just learn Kubernetes. Yes, there is a bunch of vocabulary to learn. There’s a learning curve. But I quickly found that it had the answers to my orchestration problems, where Hyper.sh or Docker Compose presented mostly limitations. Pretty early on it was clear how much more polished and robust Kubernetes was for solving the problems I had.

I also recommend working with Helm. Kubernetes feels weird because it’s an API that mostly communicates in terms of resources declared in YAML files. Helm provides a templating engine around those YAML files and commands that provide the procedural/operational logic that isn’t totally obvious when working directly with kubectl.

Plus, examining existing Helm Charts off the shelf is a great way to see how the pieces of the puzzle fit together.

Docker is great for local development. Spin up a container, mount some code to it, hack away. It’s a natural step up to docker-compose to mix additional services into your app. The docker-compose.yml format is both powerful and approachable. Now you’ve got a nascent project or proof of concept doing something interesting on your laptop.

But now you want to show someone else. Who might not have the patience to install docker, pull your repo, run your yml, etc.. You need hosting.

Hyper.sh is a smart step up that gets you a hosted instance quickly. If you’ve already gotten to some fluency with docker-compose on the commandline, then you’re ready to use hyper.sh. Their CLI utility closely mirrors how you already work with docker-compose. The documentation is solid. The web interface is informative, but really secondary to their CLI.

I haven’t used Hyper.sh for a production site yet. But there are many uses for containerized hosting other than hosting a production app, and for those cases Kubernetes is often overkill. Why give yourself the barrier of learning Kubernetes just to expose a proof of concept or dev app to collaborators and test users? I think Hyper.sh fits well for those cases.

I also tried Docker Cloud (formerly tutum) for these purposes, and didn’t enjoy it. DC doesn’t provide hosting, so hooking up with a hosting provider adds another layer of administration. The web interface purports to do everything, but was often slow. The web-based terminal app sucks. Not a fan.

Some tips:

• If you get weird quota errors, just go to the Account > Quota page and use the link to request higher quota. Seems like they’ll generally grant the increase without asking questions. If you fool around and don’t delete stopped containers and unused images, it’s easy to hit the initial quota.
• Hyper.sh doesn’t have built-in DNS, so expect to add A records somewhere else, or just access via IPs.
• Each container that needs to be accessed publicly needs an FIP (floating IP). Allocate and attach (typically via the ‘fip’ property in your yml).
• Think about using named volumes for your app, and configure those in your yml. You can hyper compose down and hyper compose up and not worry about deleting your data - volumes are maintained, and reattached to the new containers.
• If your container dies for some reason that seems inexplicable, it could be out of memory. Try increasing the container size with the ‘size’ property in your yml.
• Make sure to hyper pull [image] before you hyper compose up, if your image has been changed/rebuilt. Hyper won’t automatically check for the new version.

Cool things: they’re working on integration with log collection and a CD pipeline that starts with auto-built images.

A lot of people are waking up to the fact that it sucks to lose a game you barely knew you were playing. Donald Trump swept the board on Nov 8, and he’s already started another round. He plays with manic fervor and sneaky-good strategy. And geez, the victories the bastard walked away with. Ouch. Maybe you placed a token or two; maybe you thought he didn’t have a chance; maybe you just never thought politics was your game.

Well old son, it’s time to learn how to play.

It can be as overwhelming as an eight-hour strategy game. The board is busy as fuck. Counters everywhere you look. But it’s actually not so bad once you get a way of thinking about it. Please allow me to suggest an easy one:

Politics is kinda like Magic: The Gathering.

SLINGS AND ARROWS

Here’s the level at which most folks engage with politics: talking trash. And hey! it’s fun! You roll up to the tournament table with your buds and let your opponent know just how bad you’re gonna hand their ass back to them. Gift-wrapped. A bow on top. Everyone on your side gets the inside jokes and jibes, and that sucker doesn’t even know how bad they’re getting roasted.

Shit talk is fun.

That’s why people pass around the John Oliver clips, the small hands memes, the “omg can you believe this hypocrisy” tweets. Etc. Everything is awesome when you’re part of a team (and fuck that other team). Shit talk lets us blow off steam and process some of the bizarre and the absurd.

Talk all the smack you want, but if you’re not playing creatures and casting spells, you’ll never win a match. Meanwhile, somebody else is winning. And they probably don’t give a shit about you.

INITIAL MOVES

When you start playing politics, recognize that you’re playing with starter’s decks. When you start, don’t worry if what you play doesn’t match up that well against the opposition. You’re gonna take some lumps. That’s where learning comes from; that’s how you’ll build your deck later.

For now, remember: the most important thing is to keep playing cards.

As in Magic, politics is all about your mana pool. You can’t even mount an attack without building some base of mana to draw from. In politics, relationships are your mana. Connections are your mana. Not like “my dad’s the district attorney” connections - just connections to people. Your friends, your family. Other people who are playing politics.

If you meet one person who ends up being someone you like to work with, that’s golden.

If you can get two friends to take some meaningful political action, that’s golden.

So your initial moves are to just start personal conversations. Not facebook blasts. Get into spaces where you can meet people who are already doing work that looks interesting. Find out what support they need. Here’s just a few places you can start:

• Standing Up for Racial Justice
• 350.org
• Planned Parenthood
• Local abortion-access support funds/organizations
• Your local ACLU

YOUR BASIC CREATURES

Magic endures because there are lots of ways to play it. Taking forever to build up a ton of mana and then summoning a huge, overpowered beast isn’t a guaranteed win. Summoning basic, cheap creatures can still do damage.

And holy fuck do we ever need to do damage.

Bringing a couple friends to a protest is doing damage. Holding a house party to raise $300 is doing damage. Making a pledge with a co-worker to donate a small percentage monthly of your wages to a cool group is doing damage. It still takes mana - those connections - to know where these small actions fit in. But not really all that much of it.

Summon some basic creatures. It really matters. Sometimes a surprising amount.

Let’s take an example: hardly anybody ever talks to their city council members. Developers do. Big corporations do. Police unions do. But hardly any plain old residents. Seriously: in lots of city council elections a majority of the voters are 55 and older. And yet city councils have done cool shit: Seattle sparked a movement by passing a $15/hour minimum wage. Cities passed ranked choice voting (a way to eliminate 3rd party spoiler effect) and now a whole state is going to use it (Maine). Cities passed immigration/police separation ordinances that have Trump pissed off.

Summon some basic creatures on your city council, and you can win pretty big.

You don’t know what your opponent will play until the cards are on the table. You don’t have control over what that next card will be from the other side. The most important thing is to keep playing cards. Some of them will pan out and it will surprise the hell out of you. Look at Occupy Wall Street: a rag-tag group of radical weirdos squatting in a park. How many hundreds of protests like that went down? But that time, a chord was struck. It was the right card at the right time. They didn’t know in advance. They just kept playing cards.

DECK BUILDING

OK, Ok, at some point you do have to get past the starter decks. It helps to play the cards that give you an advantage, that fit with your style of play. A lot of times, folks are afraid to play cards, because they’re afraid of it hurting their own side. “What if I offend someone”? And it’s true, you might get called out.

If you’re not sure you’re bringing the right cards to the table, do a little reading on deck strategy. Look up intersectionality. Read a quck but powerful book on feminism or black experience in America. Look up anti-racism. There are groups of people who want to talk about these things and educate each other. We don’t want to repeat the mistakes of the past.

But nobody has the answer. That’s the important thing. Nobody actually comes to the table with the ace, unbeatable deck. It’s not your job to build one. It is your job to listen and learn. Your best strategy for deckbuilding isn’t supreme knowledge.

It’s just plain not being defensive.

Take a breath and accept criticism. It’s tough. It’s one of the toughest things, especially when you’re walking into the back room with all the folding tables and the sweaty pits for the first few times. But that’s the advanced strategy: listen, acknowledge it, say you need time to think on it. Give yourself time to get over the hurt and the rush to defend yourself.

Defensiveness doesn’t build better decks.

HIT POINTS

If you’ve read this far, then I owe you an apology: sorry. Politics isn’t actually a game. People get hurt, for real. You only have to look at the spike in post-election hate crimes, at the fear that communities feel who have been threatened by Trump and his nazi bootlickers.

Politics isn’t a game, but it is something you can lose.

So this has been a tad glib. But you know what? We gotta get through this together. It’s absurd that the weight of a Trump presidency - a Trump presidency! - sits on our shoulders. We’ve lost a bunch of hit points. Don’t sweat it. In Magic you can win from any number of hit points above zero: low doesn’t mean out.

So let’s play.