I have a confession to make. The conspiracy theorists are going to run wild with this one. The truth is my father worked in the banking industry (I promise I’m not some establishment plant sent to infiltrate the blockchain industry… isn’t that what an establishment plant would say though??). For most of my dad’s career he worked as an IT manager for some of the major banks around the world. It always seemed like a pretty sweet gig. Working on banker’s hours, he was home at 5:30 on the dot every single weekday. No late nights, no weekends…
Except for one week every year. The dreaded week that came each and every year where they would deploy all the code they’d been working on for an entire year all at once. Yes, that’s right, they would deploy everything from an entire year all at once. He’d be on-call 24 hours for days just in case something went wrong. It would always be a very stressful time. But you know what? It generally went pretty smoothly.
Now, don’t get me wrong, I’m not advocating for waterfall project management. I’m a firm believer in agile methodology. But I also think there are some important lessons we can learn from those days. The reason they deployed code so infrequently is that they had to make absolutely, 100% sure that everything worked. The startup philosophy of “move fast and break things” could not be their motto. Why?
Because their software was managing large amounts of other people’s money.
Sound familiar? They had zero room for error. And they would almost always accomplish that. How? Waterfall development emphasizes two very important things that the world of blockchain development seems to have completely forgotten about: DevOps and QA.
DevOps is the concept of managing the environment your code is running in and how you deliver that code to various environments — basically the “everything else” of writing software besides the code itself. This is a completely different skill set than computer programming. In a small startup team, many companies can get away with their lead developer handling many of these tasks, but you would never see that in mature companies — especially a mature company in the finance industry.
QA is the concept of testing that deployed code functionally and making absolutely, positively sure that everything is working as expected — in fact, during the QA process you generally want to try to break things just to see how your system will respond. If you go through the entire QA process and find nothing broken, you probably don’t know what you’re doing. This is also a highly specialized skill. You cannot expect someone who is a great computer programmer to necessarily just “fill in” on the QA side of things and expect that person to be good at it. In fact, I would argue that a programmer cannot test their own code adequately, no matter their skill level. No matter how hard they try to look at a product objectively, the person that actually built that product will subconsciously only be looking at the “happy path”. To find an error in a product you’ve created is emotionally painful. As humans, our brains jump through every hoop we can find to avoid pain. Even if that hoop is subconsciously not clicking on something we know may not completely work. You need QA people testing that aren’t afraid to hurt the developer’s feelings.
In the blockchain industry, we’ve all seen companies that have raised millions of dollars in a matter of hours. Some of them already have established teams of dozens of developers. But how often have you seen any of these teams with even one person dedicated to DevOps and even one person dedicated to QA? I’ll tell you how many times I’ve seen it: zero.
So, these blockchain companies don’t have experts on their teams in managing how their code gets to production environments; they don’t have experts on their teams to even set up those production environments; and they don’t have experts on their teams to thoroughly test these environments once they’re up and running. But what they are doing is…
Managing large amounts of other people’s money.
In the web development world, we’ve slowly but surely starting stealing some of the best parts of the waterfall methodology and integrating them into agile. The realization of the importance of DevOps and QA has exploded in the last ten years. However, all of these web developers jumping into blockchain seem to have completely forgotten all of that progress. And we’ve all seen the results. The Parity “hacks”(maybe links) were a great example. Neither of those were hacks. They were both poorly written code that a good QA person would definitely have found. Somebody from the general public then started poking around and doing something “they weren’t supposed to do.” Guess what? There will always be people doing random shit on your platform that you didn’t intend for them to do. That’s just the way the world is.
The way the industry is shaping up is to crowd source QA to the general public. This method can work for some dumb social media site or something like that. But that’s not what we’re doing in blockchain. The blockchain industry is a part of the finance industry and we need to realize there’s a reason fintech does things the way they do. Every single Dapp, every single project, every single contract is managing people’s hard-earned money. We need to mature as an industry and realize that’s the situation we’re in and stop shooting from the hip with our code deployments and testing. The methodology is out there. It works. It’s worked for decades. Commit to using it.
/rant
Oh, did I mention web3devs now offers DevOps and QA services? 🙂 Shoot us an email to discuss.