Wow, I have been working on this project for a long time. This has been a side project through-and-through: a pursuit of passion that has lived in the margins of jobs and urgent political projects. The above graphs are commits on the two biggest components of swhenever.com: the backend API and the React frontend app. You can see periods of intense activity and inactivity; some focus alternating between frontend and backend; and how much more domain logic is in the backend.

I proudly and officially released version 1.0 at the beginning of January. It’s a good time for reflections.

Domain-Driven Design worked well

Not pictured in the commit graphs: lots of hours of hashing out the domain back in 2015. I had read a lot of Implementing Domain-Driven Design and found the ideas compelling. This project was my first attempt to apply them seriously, and the team at the time (primarily Jim Cummins, Jeremy Ward and me) brainstormed and disagreed and documented domain concepts before doing serious coding. (I had a brittle proof-of-concept before that).

I think it was worth it. The resulting domain concepts (“ubiquitous language”) feature prominently in the code and have been really helpful in guiding architecture. I still think about the system in those terms even though I haven’t referenced/modified the planning documents for a couple years.

Take MVP a little more seriously

Honestly, there are parts of the system that are pretty over-engineered. I’ve cut back on aspects of it, but could go farther. Used CouchDB to store JSON documents that should have just been a longtext field in MySQL. Used RabbitMQ to serve up a couple job queues that should have just used laravel’s MySQL queue provider. It was worth the time to think through the domain and code to those abstractions, even though it was probably slower. But it was not worth the time to learn pieces of technology that will theoretically provide benefits later. Neither would have been expensive to switch to later on. That said …

I have learned so, so much

Taking up this project and sticking with it - regardless of serious stretches on the back burner - has been the best technology learning experience of my life. If I ever want to justify the slow burn or the overengineering to myself, this is it. I’ve learned Python, Docker, React, and Kubernetes. I’ve tackled the challenges of designing, building and deploying a complex system from alpha to omega. Almost three years on, sitting down to work on this is still the most rewarding and pleasurable work I’ve ever done in technology.

It’s just been an incredible blessing to make the space for a project that I deeply believe in. I’m able to do that because of a lot of privilege: specifically because I’ve worked part-time while still pulling a salary around the US household median income. Everybody deserves a life of dignity and creativity, but instead of producing that, our political economy has given all the productivity gains for 40 years to the 1%. I think about the local artists I’ve known or followed over the years and it staggers me how much more beauty the world would have in it if those folks just had a stable economic position. Artists are some of the hardest working people I know, who take some of the most risk in service of an ideal. If swhenever.com gets a little more recognition to these folks I will be very satisfied.

Slow and steady works OK

I wish I would have gotten this out to a broader audience earlier. Software that nobody is using, by definition, isn’t terribly useful. The intrinsic joys of the project are seductive: I honestly could probably have never launched anything and still really enjoy building it. But on the plus side, the system is working pretty well. It’s been working for a while. I really wanted the project to be sustainable, with as little manual data entry as possible.

It’ll be interesting if people engage with the recommendation feature. That’s the part that probably has the least basis in measureable user feedback (as opposed to my own hopes/vision), and I delayed launch to build that out. If it doesn’t get much engagement, then that would be a pretty clear example of indulging too much waterfall-style software development.

But hey, First Avenue just DM’d me, so it’s all good :)

This recording from a different show doesn’t do it justice, but holy cow it was such a joyous surprise last night when the ‘Snakes broke out a raucous version of one of my favorite Dylan shows.

Hook & Ladder is a nice little hall! I miss Patrick’s rainbow flag atop the old firehouse, but I dig it as a music venue.

One point was: don’t launch early with a convo interface that is just kind of barely working. The data you’ll get once deployed will actually be skewed by the deficiencies in your interface/bot. Instead launch a greater competence w less data, then your real data will have more meaning and drive better performance

Rubric for thinking about success:

• Smart
  • What can you do
  • How useful is it?
  • Am I training you or are you trainings me to use you?
• Trustworthy
  • How dependable?
  • How performamt?
  • How well do you know me?
• Delegate
  • I trust you
  • I can take some services for granted
  • I can focus on other decisions

Advanced Dialog attributes

• Subdialog
  • Repeat
  • Refime
  • Correct
  • Confirm,
• Mixed initiative
  • User may answer in a way that doesn’t address question, but instead expresses a desire to focus on something else
  • Be good at things related to what you’re asking user about
  • Adding constraints when replying to a req for input
• Exception handling
  • Handle exceptions to rules
• Speech acts
• Interpretation in context: partial info makes sense in context
  • Anaphora
  • World state
  • Features
• Personalization
• Proactivity
• Complex language
• Meta dialogue, talk about the dialogue itself, self awareness

State of the art is not available in a deployed toolkit. Have to search for what tools libraries to put together.

AWS Lex looks like a pretty easy way to create a conversational bot (text or audio). Very little programming required. It seems very goal oriented (define intents and inputs - slots - needed to fulfill those intents) rather than requiring you to design a big branching conversation hierarchy. Got some good notes on an architecture to support doing more sophisticated things with Lex. Limitations from the speaker: limited ability to correct Lex’s interpretations of users, cannot easily set a slot/input to be conditionally required (eg based on user session), need to route user to sub-bots in order to handle more complex/branching conversations.

If you’re not yet convinced that DevOps changes lives, this is the video to watch. Presented by the researchers behind the latest State of DevOps report. Very good practical advice about the roles of cultural/leadership changes alongside changes in technique/practice.

Research found which strategies meaningfully correlated with high IT performers, and demonstrates that high IT performance is likewise strongly correlated with higher performance on overall enterprise goals. Lots of good examples and dissection in all three areas.

One interesting claim was that they found no tradeoffs in enterprises that implemented lots of tools/processes oriented towards quality of software/deployment. Enterprises did NOT experience more failures as a result of deploying more changes. High performers had less time to release changes, higher release frequency, lower time to restore service, and lower change fail rate.

Lots of interesting points about organizational culture: pathological vs. bureaucratic vs. generative (see PDF they referenced)

Hyper.sh was great for experimentation and getting a cluster of containers up quickly. However, I ended up experiencing a number of instances where the service behaved inconsistently. I ran into more and more cases where it seemed like their basic building blocks were missing the features and glue to smoothly roll out a complex solution.

So my message to past-me is: just learn Kubernetes. Yes, there is a bunch of vocabulary to learn. There’s a learning curve. But I quickly found that it had the answers to my orchestration problems, where Hyper.sh or Docker Compose presented mostly limitations. Pretty early on it was clear how much more polished and robust Kubernetes was for solving the problems I had.

I also recommend working with Helm. Kubernetes feels weird because it’s an API that mostly communicates in terms of resources declared in YAML files. Helm provides a templating engine around those YAML files and commands that provide the procedural/operational logic that isn’t totally obvious when working directly with kubectl.

Plus, examining existing Helm Charts off the shelf is a great way to see how the pieces of the puzzle fit together.

But now you want to show someone else. Who might not have the patience to install docker, pull your repo, run your yml, etc.. You need hosting.

Hyper.sh is a smart step up that gets you a hosted instance quickly. If you’ve already gotten to some fluency with docker-compose on the commandline, then you’re ready to use hyper.sh. Their CLI utility closely mirrors how you already work with docker-compose. The documentation is solid. The web interface is informative, but really secondary to their CLI.

I haven’t used Hyper.sh for a production site yet. But there are many uses for containerized hosting other than hosting a production app, and for those cases Kubernetes is often overkill. Why give yourself the barrier of learning Kubernetes just to expose a proof of concept or dev app to collaborators and test users? I think Hyper.sh fits well for those cases.

I also tried Docker Cloud (formerly tutum) for these purposes, and didn’t enjoy it. DC doesn’t provide hosting, so hooking up with a hosting provider adds another layer of administration. The web interface purports to do everything, but was often slow. The web-based terminal app sucks. Not a fan.

Some tips:

• If you get weird quota errors, just go to the Account > Quota page and use the link to request higher quota. Seems like they’ll generally grant the increase without asking questions. If you fool around and don’t delete stopped containers and unused images, it’s easy to hit the initial quota.
• Hyper.sh doesn’t have built-in DNS, so expect to add A records somewhere else, or just access via IPs.
• Each container that needs to be accessed publicly needs an FIP (floating IP). Allocate and attach (typically via the ‘fip’ property in your yml).
• Think about using named volumes for your app, and configure those in your yml. You can hyper compose down and hyper compose up and not worry about deleting your data - volumes are maintained, and reattached to the new containers.
• If your container dies for some reason that seems inexplicable, it could be out of memory. Try increasing the container size with the ‘size’ property in your yml.
• Make sure to hyper pull [image] before you hyper compose up, if your image has been changed/rebuilt. Hyper won’t automatically check for the new version.

Cool things: they’re working on integration with log collection and a CD pipeline that starts with auto-built images.

A lot of people are waking up to the fact that it sucks to lose a game you barely knew you were playing. Donald Trump swept the board on Nov 8, and he’s already started another round. He plays with manic fervor and sneaky-good strategy. And geez, the victories the bastard walked away with. Ouch. Maybe you placed a token or two; maybe you thought he didn’t have a chance; maybe you just never thought politics was your game.

Well old son, it’s time to learn how to play.

It can be as overwhelming as an eight-hour strategy game. The board is busy as fuck. Counters everywhere you look. But it’s actually not so bad once you get a way of thinking about it. Please allow me to suggest an easy one:

Politics is kinda like Magic: The Gathering.

SLINGS AND ARROWS

Here’s the level at which most folks engage with politics: talking trash. And hey! it’s fun! You roll up to the tournament table with your buds and let your opponent know just how bad you’re gonna hand their ass back to them. Gift-wrapped. A bow on top. Everyone on your side gets the inside jokes and jibes, and that sucker doesn’t even know how bad they’re getting roasted.

Shit talk is fun.

That’s why people pass around the John Oliver clips, the small hands memes, the “omg can you believe this hypocrisy” tweets. Etc. Everything is awesome when you’re part of a team (and fuck that other team). Shit talk lets us blow off steam and process some of the bizarre and the absurd.

Talk all the smack you want, but if you’re not playing creatures and casting spells, you’ll never win a match. Meanwhile, somebody else is winning. And they probably don’t give a shit about you.

INITIAL MOVES

When you start playing politics, recognize that you’re playing with starter’s decks. When you start, don’t worry if what you play doesn’t match up that well against the opposition. You’re gonna take some lumps. That’s where learning comes from; that’s how you’ll build your deck later.

For now, remember: the most important thing is to keep playing cards.

As in Magic, politics is all about your mana pool. You can’t even mount an attack without building some base of mana to draw from. In politics, relationships are your mana. Connections are your mana. Not like “my dad’s the district attorney” connections - just connections to people. Your friends, your family. Other people who are playing politics.

If you meet one person who ends up being someone you like to work with, that’s golden.

If you can get two friends to take some meaningful political action, that’s golden.

So your initial moves are to just start personal conversations. Not facebook blasts. Get into spaces where you can meet people who are already doing work that looks interesting. Find out what support they need. Here’s just a few places you can start:

• Standing Up for Racial Justice
• 350.org
• Planned Parenthood
• Local abortion-access support funds/organizations
• Your local ACLU

YOUR BASIC CREATURES

Magic endures because there are lots of ways to play it. Taking forever to build up a ton of mana and then summoning a huge, overpowered beast isn’t a guaranteed win. Summoning basic, cheap creatures can still do damage.

And holy fuck do we ever need to do damage.

Bringing a couple friends to a protest is doing damage. Holding a house party to raise $300 is doing damage. Making a pledge with a co-worker to donate a small percentage monthly of your wages to a cool group is doing damage. It still takes mana - those connections - to know where these small actions fit in. But not really all that much of it.

Summon some basic creatures. It really matters. Sometimes a surprising amount.

Let’s take an example: hardly anybody ever talks to their city council members. Developers do. Big corporations do. Police unions do. But hardly any plain old residents. Seriously: in lots of city council elections a majority of the voters are 55 and older. And yet city councils have done cool shit: Seattle sparked a movement by passing a $15/hour minimum wage. Cities passed ranked choice voting (a way to eliminate 3rd party spoiler effect) and now a whole state is going to use it (Maine). Cities passed immigration/police separation ordinances that have Trump pissed off.

Summon some basic creatures on your city council, and you can win pretty big.

You don’t know what your opponent will play until the cards are on the table. You don’t have control over what that next card will be from the other side. The most important thing is to keep playing cards. Some of them will pan out and it will surprise the hell out of you. Look at Occupy Wall Street: a rag-tag group of radical weirdos squatting in a park. How many hundreds of protests like that went down? But that time, a chord was struck. It was the right card at the right time. They didn’t know in advance. They just kept playing cards.

DECK BUILDING

OK, Ok, at some point you do have to get past the starter decks. It helps to play the cards that give you an advantage, that fit with your style of play. A lot of times, folks are afraid to play cards, because they’re afraid of it hurting their own side. “What if I offend someone”? And it’s true, you might get called out.

If you’re not sure you’re bringing the right cards to the table, do a little reading on deck strategy. Look up intersectionality. Read a quck but powerful book on feminism or black experience in America. Look up anti-racism. There are groups of people who want to talk about these things and educate each other. We don’t want to repeat the mistakes of the past.

But nobody has the answer. That’s the important thing. Nobody actually comes to the table with the ace, unbeatable deck. It’s not your job to build one. It is your job to listen and learn. Your best strategy for deckbuilding isn’t supreme knowledge.

It’s just plain not being defensive.

Take a breath and accept criticism. It’s tough. It’s one of the toughest things, especially when you’re walking into the back room with all the folding tables and the sweaty pits for the first few times. But that’s the advanced strategy: listen, acknowledge it, say you need time to think on it. Give yourself time to get over the hurt and the rush to defend yourself.

Defensiveness doesn’t build better decks.

HIT POINTS

If you’ve read this far, then I owe you an apology: sorry. Politics isn’t actually a game. People get hurt, for real. You only have to look at the spike in post-election hate crimes, at the fear that communities feel who have been threatened by Trump and his nazi bootlickers.

Politics isn’t a game, but it is something you can lose.

So this has been a tad glib. But you know what? We gotta get through this together. It’s absurd that the weight of a Trump presidency - a Trump presidency! - sits on our shoulders. We’ve lost a bunch of hit points. Don’t sweat it. In Magic you can win from any number of hit points above zero: low doesn’t mean out.

So let’s play.

Some of my starting points, off the top of my head:

• People are creative
• People care about each other
• People want to succeed together
• Flat means no capricious authority
• Flat means you decide how you work
• Flat means you have a say in the definition of success
• Flat means you own outcomes, good or bad
• Flat means you solve problems instead of being stuck with them
• Flat means your co-workers matter
• Flat means learning
• Flat means higher expectations
• Flat means nothing is safe from questions
• Flat means decisions take longer
• Flat means decisions are better
• Flat means work is challenging
• Flat means work is meaningful

But not for everyone. I’m lucky to have had years of experience working in flat environments. While many folks are steeped in conventional management and hierarchy, many are not. It’s worth observing that people bring different expectations into a workplace, and that no management style is frictionless for 100% of the people in an enterprise. Here are a few experiences that oriented me with flatness:

• Starting my own business. I started a tiny web development/consulting company with my best friend when we turned 18. Went right from college to doing it full-time, figuring out taxes, making decisions about clients and contracts and rates. Hiring (and sometimes firing) a small group of employees. Stuck with it and paid the bills for almost ten years.

• Running for office. I ran for the MN House of Representatives in 2006. That was an amazing experience for a 23-year-old. There is nothing like knocking on doors as a candidate to get the feel for just how precious it is for another person to give you a few minutes to listen.

• Co-founding a non-profit and serving on various all-volunter committees. Motivation and accountability are huge challenges when working with volunteers. But in some ways, I think the paycheck and regular hours can mask just how much these are challenges with paid employees as well.

I am really thankful for these experiences. A few things that strike me, as I reflect on them:

• Failure and unpredictability. It strikes me that I’ve failed to achieve my goals at least as often as I’ve succeeded, despite spending equal amounts of time and energy in all cases. I think I’ve learned that you don’t have to be perfect, and you don’t have to be an expert. Nobody can predict the future. Small business and grassroots politics are both risky endeavors in environments where many of the variables are out of your control. Some humility about what I can actually affect, I think, paradoxically makes it easier for me to go out on a limb.

• Working flat isn’t that different from entrepreneurship. Starting your own business entails a lot of the same challenges as working in a flat organization. Success depends on your initiative and there’s no one else to blame if it doesn’t happen. It’s funny that the entrepreneur is so lionized in this culture, while at the same time strict hierarchy is regarded as a norm for employees.

• All-volunteer projects are great proving grounds. If anyone can just decide to stop showing up, you’ve got to constantly be persuading and motivating the people around you. And finding the ways to work that motivate yourself!

• Success is that much sweeter, and even failure means something special. The worst scenario I can imagine working in is where nothing I do matters. For better or for worse. All of these experiences have meant a lot to me. They changed me in important ways. I think having the sense of ownership that comes with initiative, ownership and collaboration is a big reason why I’ve gotten so much of them.

  I think those are some of the things people need in order to be successful and comfortable in a flat org: opportunities to be trusted, the experience of failure under your own terms, and the chance to try again with what you’ve learned.

“To comport oneself with perfect propriety in Polygonal society, one ought to be a Polygon oneself”
Flatland: A Romance of Many Dimensions

Edwin Abbott’s story is a satirical riff on rigid hierarchy. It’s a literal flat land – as in two-dimensional – but flatlander society is anything but. The inhabitants are grouped into immutable social castes according to their number of sides. But the inhabitants only see each others edges, within the plane they share. Never from above. Only a careful study in recognizing the subtleties of distance allows a hexagon to be distinguished from an octagon. The higher classes are granted this study, while the lower classes are left to hide their confusion behind a blanket deference to their betters.

I think one of the common concerns raised about flat organization is that it is confusing and chaotic. Anyone can be responsible for anything? How is stuff going to get done? Who do I ask about X? How do we make a decision if we don’t know who is “supposed” to make it? How do I know when I can say “no” and when I should compromise?

These are all legitimate questions! Flat organization is strange and confusing, if you’re inexperienced in it. But I ask you to wonder about the conventional, hierarchical organization that feels more comfortable to you. Think about the times when you’ve just gone along with a decision you thought was dumb, just because somebody higher up said so. Think about the times when you were doing work that you know was unchallenging and rote, just because that was your job. Think about the times you had to complete a task in a completely pointless, wasteful way, just because the person who made the requirements doesn’t actually know the work, at all.

How did you cope with those situations? How did you avoid getting too frustrated or confused with your lack of agency to change them? How do we accept them day in, day out? I suggest that it’s simple: we’re raised in hierarchical organization. We’re trained to navigate hierarchy deftly, to “climb the ladder.” We’re taught to accept the inefficiencies and dissatisfaction as natural, and to focus only outcomes like sales, salary and promotions.

In a polygonal society, we’re all polygons. When a sphere visits Flatland, the (square) protagonist is utterly baffled by the notions of three-dimensional space. The novel is a classic expression of the basic sociological impulse: to consider that we might seem just as strange through the eyes of another society as they seem to us.

A little dose of that thinking is necessary when making the leap to flat organization. It means suspending the immediate impulse to dismiss practices as strange, and to be willing to question some of what we have taken for granted as normal about conventional, hierarchical organizations. It’s not always easy or straightforward work.

Of course, the onus is on flatvocates to demonstrate that the benefits of flatness are worth the growing pains.