The seven deadly traits of startup culture
Things aren't going well. You struggle to understand how they got so out of control. Everything pointed in the right direction at start: you ticked all the right the boxes and made all the right choices. Yet technology fails to comply. You have agile, containers, CI/CD, and a cloud bill. But things still break more often than desired and fixing them takes more work than it was supposed to. You are swamped with help requests. While you're fixing one thing, something else breaks. Maintenance is chocking your creativity. Every day is met with a stress loop. You have no time to build something of value.
You've finally decided to stop all work for a day to just look into the situation and you've diagnosed some of the problems at surface level. Could there be a deeper structural cause?
I don’t know if this description is familiar to every reader. I've seen the same pattern over and over again across different projects, but it took me quite some time to synthesize it. My educated guess is that at the root of a hurdle of this kind lie the seven deadly traits of startup culture, or at least some of them:
1. newer is better 2. innovation must be present at every layer 3. complex is better 4. trendy is better 5. buy is better than build / build is better than buy 6. temporary things are fine 7. decisions must be reached democratically within the team
Obviously, if all of these traits check, you run a significant risk of building a Rude Goldberg machine designed by committee.
I've written about mode-1 thinking before. I've also written about hype trains and ideologies. I went back to read those texts. The empirically established traits mentioned above seem to build on the same background.
A pragmatic engineer would pull his hair out if told to work in an environment such as the one described. But nobody is ever told. Problems emerge as time goes by and their root causes aren't always fully understood.
If these traits are a real concern, what alternative cultural traits should we foster? Here is my working hypothesis:
1. reliable is better 2. one must choose carefully at which layer(s) to innovate 3. simple is better 4. trends don't matter 5. buy vs build depends on the context 6. temporary things are NOT fine 7. the manager listens to the team and then decides, embracing risk and accountability
Let’s dive into each point.
Reliable is better: this reminds me of the classic Hot / Crazy scale post by Stackoverflow founder Jeff Atwood. It’s fun to have hot-crazy things in your bedroom computer, but not as much fun to see an entire company depend on them and having them break during your holidays. I suspect that people pushing for newer technology don't expect it to be unreliable… but balancing novelty and reliability is what a manager is meant to to. Was optimism ever a good strategy?
One must carefully choose at which layer(s) to innovate: if a startup is building a disruptive idea perhaps it should start by building it on top of proven infrastructure technology; if a startup is building game-changing performance over an already existing concept it will need to be edgy on infrastructure technology to make sure that the best performance is delivered; innovating on both layers is also possible, but very very hard and therefore less likely to succeed. Innovation is sometimes overrated.
Simple is better: the issue of complexity has been discussed publicly discussed many times but its importance doesn't seem to come across. The complexity fetish is very real. Excessive complexity can crush an entire company due to the unsustainable amount of maintenance it requires. Usually, things look pretty on a PoC, but at some point the first problems arise. You have to do security updates. You have to do bugfix updates. Something subtle sneaks in and breaks production. Debugging is hard. Your time is short. Your team is small. you are not Google. Complexity should be justified by clearly stated business requirements and backed by a proportional allocation of maintenance hours.
Trends don't matter: some trends are good, other trends are bad. None of them matter because real Engineers are not fashion victims. One must choose what best suits business requirements, not what big tech marketeers are banging the drum about.
Buy vs build depends on the context: there is a classic tension between buying and building. This is completely normal. By building you gain control at the cost of insourcing a concern. By buying you give up control while, to a certain extent, you outsource a concern. In both cases you'll pay: either for time that building takes or for the service that is provided to you. Buying something doesn’t fully outsource the corresponding concern because there will always be integration issues (dispersed authentication sources anyone? dependency on 3rd party APIs that often fail or change over time?) The buy versus build decision is a trade off for which context is of paramount importance.
Needless to say, you should never outsource your core business. More often than not, you shouldn't outsource things that are very near your core business. But you should not rebuild existing commodities either, unless you have a business-backed reason to do so.
Temporary things are NOT fine: temporary things are one of the biggest lies in the IT industry. Nothing that goes to production is ever temporary in the short term. And once in production changing anything requires a tight fist, and nevertheless comes at the cost of annoying whoever depends on it. Please don’t. Just don’t. Unless by temporary you mean “for 5 years”, which is fine, since 5 years is a reasonable lifespan for a product. In that case, temporary means definitive so we're good. If you don’t mean “for 5 years” have in mind that it will probably be for 5 years anyway. Stop fooling yourself.
The manager listens to the team and then decides, embracing risk and accountability: last but not least, we need to address the endemic I don't decide, the team decides, a common behaviour found in incompetent project managers who don't want to be held accountable for the consequences of their decisions. Project managers should listen to their teams, but a consensus can't always be reached. And even if a consensus is reached, it might not make sense. There's a hierarchy in place because some people are more experienced than others, and they have specific skills related to management. A project manager must be competent in the subject practiced by the team and must be able to reach sounds decisions when the team is unable to do so. That’s how you spare a company several years of suffering as consequence of a poor decision. Not convinced? Ask Steve Jobs about democracy and product management.
Now that we've landed the theoretical concepts, let’s have fun with some practical dilemmas:
- LTS distribution with stable reproducible python packages and formal security updates or the latest Pip modules on top of an informal docker image?
- 2 large virtual machines for redundancy or the infinite scalability of Kubernetes?
- monolithic project shared by a small team or several micro services with independent repos and CI/CD pipelines?
- HTMLX or React?
The answer is, obviously, IT depends, and thinking about how context might influence these decisions is an exercise for the reader. If you came up with a different answer irrespective of the context, including the now classic the team decides, you might want to resume from the top of the post.