Subvisual | The Worst Decision Of My Career

This is a reflection on software development and complexity. Let’s start with
some quotes to make me look smart:

A complex system that works is invariably found to have evolved from a simple
system that worked. A complex system designed from scratch never works and
cannot be patched up to make it work. You have to start over with a working
simple system — Gall’s law

I only came across this quote recently, but I think that it summarizes what
I’ve learned these past seven years. Another idea that I’ve found in a few books
is that programming in isolation is problem-solving, but software engineering
is all about managing complexity. This distinction between programming and
software engineering makes sense to me, probably because of my mental models.
I’m sure we can all disagree on this, but that’s not the topic of today.
Complexity is.

Complexity

Almost everyone I know in the software industry wants to build products that
solve real and challenging problems. Change the world! The last thing you want
is to build another CRUD application. They are fine, but it gets boring after
a while.

Most of us will get bored without novelty. That’s why the software industry has
a recycling mechanism to keep things fresh and interesting for us: every once
in a while a new, or different, language/framework will rise and light the path
for a brighter future, overthrowing the existing standard, demanding that we
learn it to be on top of the game.

We change our tools, but we keep building the same things over and over. I’ve
been doing this long enough to see that we are just going in circles. Let’s be
honest, most of us aren’t building life-changing products that wouldn’t have
been possible ten years ago, so let’s not jump straight into another shiny
technology that just came out and is going to solve all of our problems.

Solutions, solutions, solutions

For a while, the solution to every problem I encountered was a Rails app. —
this one is mine

Over the years, I’ve seen companies rewrite their products to change languages,
frameworks, or architectures because they were told that the change would make
their product better, run faster, and scale. By the way, you’re only a true
Ruby developer after you’ve heard the question “but does it scale?” at least
one hundred times (kidding, but it’ll happen).

Think about it, how many times were you sold a new programming language,
technology, or a concept like microservices, serverless, event sourcing, clean
architecture, micro frontends. There are many preachers in the software world.

At Subvisual, we started using Elixir a lot these past years, so I’ve learned
a bit about the history of Erlang. If you don’t know, Erlang uses the Actor
Model, and the interesting part is that the designers of Erlang only learned
about the Actor Model after having designed Erlang. This is important because
the designers of Erlang didn’t start with the intent of applying the Actor
Model; it came as a solution to fault-tolerant distributed programming.

If all you have is a hammer, everything looks like a nail Abraham Maslow

The designers of Erlang picked the right tool for the job and most of us want
to do the same. Unfortunately, there’s usually more than one “tool” that would
be “right,” but to know which ones, you need to know what “job” you’re solving.
All of us should know this, but I keep learning about teams that fail to do
it. There are so many companies, with a handful of experienced developers, that
don’t know what they are building, don’t have a clear business yet, but their
product is already built on complex architectures and technologies such as
microservices, Kubernetes or event-sourcing.

Improving

Every project we start from scratch is an opportunity to do it right. We’ll
think to ourselves: This time I won’t fall into the same traps! I have learned
my lessons, I’ve studied the books, and I even met some of my gurus that wrote
them! This time I’ll follow “industry standards”!

I am the “architect”; I will design the perfect system! Without me, none of
this will be possible. Those mindless programmers have no idea what they are
doing. I, and only I, am the true heir of Martin Fowler!

This was a bit dramatic, but I needed a break from all of that whining. I know
it’s easy to fall into these traps. It’s even easier when your company raised
money, and everyone is expecting you to deliver the absolute best product ever
because they are paying you for it! This is why your starting team is so
important; they have to withstand the pressure. They must know that to build
the grand vision, they have to go one step at a time.

The decision

I’ve made many bad decisions, and I’ve been fortunate enough to suffer the
consequences of those decisions. A lot of developers don’t get to experience
consequences, so they never learn.

So what was the worst decision of my career? I don’t know. But the title of
this blog post is inspired by something an old colleague said. At the time, we
were working for a product company that reached for event-driven architecture
and event-sourcing too soon. They knew little about their market, and the
choices the software team made were crippling their ability to change. Business
rules were almost set in stone. Migrating data was a pain and the source of
many bugs. And unfortunately, because the team didn’t have experience with
event-sourcing, the event store, which kept the state of all services, was also
being used as an event-bus to communicate between services. Because of that, it
was possible to couple one service to the internal state of another, which
happened a lot. This almost invisible coupling made everything worse.

What did we do against such an unpredictable system? We took it apart: merging
services that were too coupled; defining clear boundaries between services;
moving some services away from the event-store into a traditional database;
making some communication channels synchronous; writing integration and
end-to-end tests.

When we were finished it was still an unnecessarily complex system, but it was
one that we could change with some confidence. After that, we defined
a long-term plan for the product’s architecture and technology, but we didn’t
implement it. We waited for the right moment when something was starting to
slow us down to make a small step in that direction. When the business goals
changed, we changed our long-term plan, and once again made small steps in that
direction when we felt the need for it.

Eventually, complexity found its way again into the codebase, but it was fine
because the codebase was evolving slowing, adding and removing complexity when
necessary.

Making the right call

Was that the worst decision of his career? I don’t think so. The issue with him
going for event-sourcing and event-driven architecture was that it wasn’t the
right moment, but my colleague didn’t know that. He thought the goals for the
next years were well defined, but unfortunately, they never are.

Should you use microservices or event-sourcing? Maybe, it depends on the
context and the client. The decisions I make are the best that I can with the
information that I have. For instance, when I’m part of the team that’s
starting a product, the technology we pick will depend on how we’ll hire: if
you want to build an office in Portugal, we have to make sure we have
developers for that technology available. We have to think things through, and
some things you only learn from experience. This applies to the systems we
design as well. I’ve seen enough people design around what they believe the
product will become in two years to know that those designs always fail to
accommodate the changes that will come. So we design systems for small,
incremental changes. Do the smallest thing that will get us started and
doesn’t compromise our ability to change once we know what the business needs.

Read More

54 Replies to “Subvisual | The Worst Decision Of My Career“

  1. I have been surfing online more than 3 hours lately, but I never discovered any attention-grabbing article like yours.
    It is pretty value enough for me. Personally, if all web owners
    and bloggers made just right content as you probably did, the web can be a lot more helpful than ever before.

  2. Hey there would you mind letting me know which hosting company you’re
    using? I’ve loaded your blog in 3 completely different
    browsers and I must say this blog loads a lot quicker then most.
    Can you suggest a good internet hosting provider
    at a reasonable price? Kudos, I appreciate it!

  3. I’ll immediately seize your rss as I can not
    find your email subscription link or e-newsletter service.
    Do you’ve any? Please allow me recognize in order that I may subscribe.
    Thanks.

  4. Thank you for some other great post. The place else may anybody get that type of information in such a perfect approach of writing?
    I’ve a presentation subsequent week, and I’m on the search for
    such info.

  5. Also, here’s a short, sample book report that is not about a college-level novel, but does a
    good job of illustrating the mix of summary and critical analysis
    that should go into a college level book report or review.

  6. Delighted report concerning %BT%? It sounds so great in becoming a aspect of it and in addition to comment on it!
    Let me get the full story info on the idea in my website!

  7. It’s perfect time to make some plans for the longer term
    and it is time to be happy. I have learn this post and if I may just I wish to
    suggest you some attention-grabbing issues or tips.

    Maybe you could write next articles regarding this article.
    I want to read even more things approximately it!

  8. I have been browsing online more than 2 hours today,
    yet I never found any interesting article like yours. It is
    pretty worth enough for me. In my view, if all site
    owners and bloggers made good content as you did, the web will be much more useful than ever before.

  9. It is my belief that mesothelioma is actually the most fatal cancer. It has unusual traits. The more I look at it the more I am confident it does not behave like a true solid human cancer. When mesothelioma is a rogue viral infection, therefore there is the prospects for developing a vaccine and also offering vaccination for asbestos exposed people who are at high risk of developing long run asbestos associated malignancies. Thanks for giving your ideas on this important ailment.

  10. Its such as you learn my thoughts! You seem to understand a lot approximately this, such as you wrote
    the ebook in it or something. I think that you just could do with
    a few p.c. to pressure the message house
    a bit, however instead of that, that is great blog. A fantastic read.
    I’ll definitely be back.

  11. I’ll right away seize your rss as I can’t find your email subscription hyperlink or e-newsletter service.

    Do you have any? Please let me realize so that I may subscribe.
    Thanks.

Leave a Reply

Your email address will not be published.