Four years ago I started working on a BigProduct (13+-year-old platform, thousands of users, thousands of developers) at a BigCorp (35k+ employees), as a software engineer.

It was quite a change for me, coming from my previous experience - 7 years at a small-ish & lean service company that did mostly project-oriented work for startups.

Here is some of the stuff I learned along the way.

Complexity is king

To be honest, at the very beginning, the overall velocity of product development looked quite slow to me. I was wondering why everything moves so slowly.

It didn’t take much until I started realizing the complexity of the whole thing. And that sometimes being able to ship anything in time is sort of a miracle.

You see, when you have N moving parts, the complexity is very rarely O(N), because these moving parts start having certain connections. Most of the time, the complexity is more like O(N^2). Being able to scale sort of linearly takes an enormous, constant architectural effort. And is not always possible or viable.

You need to support a million use cases on a complex stack, while not breaking use cases for critical customers and preserving backward compatibility. Sorry, you can’t cowboy-code your way through it anymore. That was an important lesson, related to the next one…

Shortcuts eventually bite you hard

You didn’t try that painful-to-test obscure use case properly, thinking nobody would go through that code path anyway? Bad news, someone from the thousands of users decided to do exactly that. And this broke an important flow. And tech support wasn’t able to solve the problem.

When you don’t do demo-driven-development for small startups, but work on a product that’s used by a lot of people, your mistakes would eventually end up back in your basket. So you better think things through before shipping your code.

Pushing the pendulum to the other extreme and endlessly bike-shedding on every trivial thing isn’t an answer either, as it will kill your productivity. The question “Is this a critical thing?” is not always trivial

Tech leads solve people problems

At the beginning of my career, I thought that being the best rock-star coder makes you the perfect tech lead. And figuring out the right person for this position was easy - just pick the best coder. Yes, I was that silly.

Then I realized that senior engineers (staff+) need to deal with increasingly people-oriented problems. And that in surprisingly many situations, the technical solution is easy to hack, whereas being able to sell it to all the stakeholders (who have different agendas and sometimes even business priorities than you) is the challenging part that involves a lot of interpersonal and communication skills.

Don’t get me wrong - of course, being a rock star coder would help tremendously and having solid competence is a prerequisite after a certain point. But don’t expect to have much success at higher individual contributor levels by relying only on that.

Testing pays off

Having worked mostly in small-ish teams (where you are quite literally next to each other) and for startups (where the customer might be just a few sprints away from running out of money), I wasn’t accustomed to heavy testing, especially the higher levels of the pyramid - integration or E2E testing. Investing too many resources in testing is not always an easy sell to stakeholders for a small project.

But if you want to sustain a heavily-used product for years, without getting overwhelmed by angry users, you need to make a real investment in testing.

Unit testing is nowadays an industry standard. But I’ve never been in a team with so big coverage by automated E2E tests that run 24/7, on different platforms and browsers. This requires some effort, but regressions are caught very early. Also, stuff like visual testing is like the cherry on top. Now you can upgrade your UI frameworks and have confidence that you haven’t broken half of the application.

Internal documentation pays off

Small projects don’t always have enough resources to spend on stuff that is not directly related to product development, see the previous section about testing. This affects documentation too.

However, when you work in a product team where projects last basically forever and teams are changing constantly (sometimes nobody from the original developers is still around), documenting is critical. And I don’t mean just maintaining 3 internal wiki pages, but writing everything down. Some examples:

• Internal docs for the team - guides, how-tos, glossaries.
• Architecture documents - the overall big picture of the system. It’s even better if there’s a process for reviews/signoffs from more senior folks or other stakeholders. Such a process will polish things a lot, leaving an excellent “paper” trail to figure out why the hell things work that way.
• Functional specs for features you implement - documents describing how and why things behave that way. Signed off by all stakeholders.
• Good commit messages - this is often overlooked but helps enormously when dealing with things on the lowest level, e.g. strangely-looking hacks.

Ideas are cheap (most of the time)

During my university years, I had a big appreciation for “idea generators” - these people that have the “Eureka!” moments, giving great ideas for the world to develop. That looked like magic to me, probably because I’ve always been worse at strategy/ideas than at operation/execution.

At the company, like many others, there are a lot of opportunities for implementing your ideas during your regular work time - e.g. internal hackathons. And execution matters a lot - you might have the best idea in the universe, but you need to implement something meaningful to “sell” it internally. I couldn’t quite get that part - aren’t the ideas the most important thing, after all?

Then I started realizing the Complexity (see above) and the big picture started to clarify. At that point, you have many people working on a big product. Almost everyone has all kinds of ideas for improvements. Some of these ideas are quite obvious. Most of them are good. New people come into your team and wonder “Why you guys haven’t considered XYZ?”. 90% of the time, it has been considered in the past, and it has turned out this wasn’t the sort of improvement you could implement in an afternoon. Or a week or two. Instead, it would require a lot of focused effort.

Lesson learned - you need to own your ideas, otherwise they’re mostly useless. And the really hard thing is not the idea itself, but making something meaningful of it - prototyping, validating, pitching, measuring, iterating again, etc.

Still, this point has exceptions. Some people are naturally better at ideas than at execution. They have a vision of how the pieces in the puzzle will fit (which is a critical thing) but do not always have the bandwidth to implement it. Others take the baton and implement great stuff.

Change is constant

Reorgs, charter changes, new priorities, new initiatives. That stuff happens all the time. And was kind of a surprise for me, as I expected things in such structures to be pretty static.

You need to be adaptable and deal with change.

Like it or not, politics are part of the game

Someone said, “Politics are bad only when you’re on the losing side”. Even though I’m not happy with that and wish things didn’t work that way, I tend to agree.

When a lot of people need to collaborate on something, politics are an inevitable part of the game. And not in a primal “Game of Thrones” way, but more of a “let’s try to persuade this completely unrelated team/person in a completely different part of the org chart, with completely different priorities and agenda, to do something for us”.

The lesson: if you want to be successful, being able to navigate through these waters is important. Having a manager that is not just a proxy, but can position your team well enough is also extremely helpful.

Of course, it’s easy to get overly cynical and end up in a hopeless rabbit hole where you see a huge Kafkaesque system that works in a completely mystical way and cannot be swayed in any way.