Entropy and Inertia in Engineering Leadership
Entropy and Inertia in Engineering Leadership
Navigating the Inevitable Laws of Organization
One of the challenges for engineering leaders is that they cannot see the results of their work immediately. When they were individual contributors, the code they wrote provided rapid results. They could interact with it or see computed outcomes.
Many engineers are just a few clicks away from making their work available to thousands, if not millions, of users. However, when an engineer becomes a leader, things change. Sometimes, it takes weeks, if not months, to see the effect of their work. And their passiveness in certain areas slowly leads to quality deterioration, mess, and disorder.
There are two physical forces that stand behind this challenge. These are inertia and entropy.
Inertia
Inertia is the tendency of an object to "keep on doing what it's doing." It's the resistance to change - like a supertanker that can take an extra mile to change direction or stop.
Many engineering leaders experience organizational inertia in a hard way. They think they know what the good looks like. They have seen solutions to their team's problems at conferences, in books, online articles, or in their past jobs.
The problem is - the team may not want these changes. In my experience, when I introduced stability metrics to drive decisions with data, the team felt observed and blamed for their failures. They preferred to solve their problems quietly, without drawing attention.
Inertia in an engineering organization is the tendency to stick with existing processes, technologies, and tools, even if they are no longer the best fit for the current situation.
It can be caused by several factors, such as:
Cognitive Biases and Habits: Our brains are wired to favor familiar patterns and resist change because it burns much less energy to work on "autopilot" (I highly recommend reading "The Power of Habit" by Charles Duhigg, which explores this subject).
Organization's Legacy: Companies often become resistant to change due to bureaucratic structures, vested interests, and a fear of disrupting established practices. The adage "If it ain't broke, don't fix it" applies here.
Culture: Norms, values, and beliefs become deeply entrenched in us or an organization's nervous system. "The way we do things made us who we are and where we are today," is often said by the old guard (or worse - their ghosts, since they are long gone from the company).
Inertia can be dangerous in a tech company, hindering progress and innovation.
According to the Four Factors Essential to the Success of Any Product, we must enable growth and expansion, cut costs, and improve customer satisfaction.
For engineering teams, this means we need to:
Increase our agility: Constantly look for tools and technologies that make our work more efficient.
Decrease costs: Optimize our systems and solutions so our costs don't increase as fast as our user base or product portfolio.
Improve quality: With the growing complexity of our systems, we need to not only maintain their quality but also constantly improve it.
Generalize things: Ensure that we build general systems, not singular features, so things that took weeks to build today should take days or hours.
None of these is possible without constant innovation in the way we work.
Disrupt or be disrupted.
But why is all of that? Can't we just continue what we're doing for an extended time period to achieve our goals without constantly disrupting and innovating? If this is only inertia we fight with, this could work.
But…
Entropy
There is another force playing a role in making our work harder - it's entropy. In physics, it measures the degree of disorder in physical systems. And according to the second law of thermodynamics, entropy always increases.
Organizational entropy causes companies to become disorganized and unfocused over time. It also makes our software increasingly complex, declining in quality, and more difficult to maintain.
It's due to a few factors:
All Aspects of Growth: Organizations get bigger, and products become more complex, as does the technology behind them. One of the non-obvious reasons for this is that it's easier for us to add things rather than take something away (see this Nature article: Less is More: Why Our Brains Struggle to Subtract). When things get bigger, they become harder to manage.
Bureaucracy: As organizations become more bureaucratic, decision-making can become bogged down. Endless chains of approval, clashing priorities, and dependencies between siloed teams don't help.
The Accumulation of Technical Debt: Our software becomes more complex and outdated, the documentation doesn't follow changes, and architecture is not being adjusted to changing needs.
Loss of Focus: Things fade out. The company's mission, vision, values, and principles - if not repeated endlessly or expressed in actions, lose their power.
Entropy cannot be ignored - unaddressed, it slows us down and causes things to decay. For engineering teams, it often means a decline in team agility, increased costs of running and keeping the system healthy, and a focus on maintenance rather than value addition. In extreme cases, unaddressed entropy can paralyze an organization and stop it from moving forward. Many leaders have witnessed this:
Old frameworks or databases, which are nearly impossible to update in one shot,
Performance limits of our systems that stop us from processing more data, new clients, etc.
Tangled dependencies and monolithic systems that are too complex to handle one more functionality that needs to be added.
Overcoming Entropy and Inertia in Your Work
As an engineering leader, it's your job to fight entropy and inertia. Here are some inspirations:
(Inertia) Leader's discipline - changes take time. You need to show up, be persistent, and overcommunicate. Never assume that things said once will work. Especially when changing people's habits, culture, and rooted processes, you need to repeat them endlessly, bring additional context, listen to feedback, and address concerns. If you want to maximize your impact and focus your discipline on the right things, check out the Leader's Week framework.
(Inertia) Use crisis as an opportunity to change an organization resistant to change. Inspiration for you: A Crisis is an Opportunity for Change.
(Entropy) Engineer's discipline - prioritize writing quality code, which is tested from the beginning of its existence, keep code documentation up-to-date, and define the architecture or principles you will follow. You can use the concept of Principles to set your standards for the team.
(Entropy) Work cautiously with technical debt - You can use Ten Types of Technical Debt as a base framework to stay ahead of it. If not the technology but rather the human aspects of your leadership are at risk, check Top Ten Factors of Developers' Productivity.
End words
Understanding entropy and inertia in the context of organizational dynamics is more than an intellectual exercise. It's a call to action. Leaders who recognize and address these forces can steer their teams through the ever-changing seas of technological and market demands, ensuring not just survival but thriving innovation and growth.