TECC 268: How Engineers Can Manage Complexity in Their Organizations

In this episode, we talk to Jeff Doolittle, a software architect and transformational leader with more than 20 years of experience designing and architecting software systems about the importance of complexity in engineering, and how engineers and engineering leaders can manage complexity in their organizations. Engineering Quotes: Here Are Some of the Key Points Discussed in This Episode: Engineers have a better grasp of complexity than most others do because they have been exposed to mathematics and real-world constraints. If you have a system with seven components, and those components can interact with each other without constraints, you will have a system that has 5,040 possible combinations and permutations of the seven components. This is where the complexity comes in because people can only have four to seven different things in their mind at one time, and do not inherently understand things like large numbers. Most software systems today have a lot more than 14 micro-services, which gives them over 87 billion possible combinations. The challenge of complexity is not only a software problem — it’s a human challenge too. Technological complexity is excessively reliant on technologies and tools, especially new and shiny ones. If we are obsessed with our tools or technologies and want them to work in vast and different ways, you will get a system that has a lot of technological complexity. Integration complexity is where you have an overabundance of components that are connected together with a lack of consistency, cohesion, and constraints. It causes micro-service mayhem — many parts that are thrown together, making it unmaintainable. Organization complexity results when managers do not filter the new information they receive, and the engineers who work for them therefore discuss all variations of the information with each other. It causes the organizational structure to mimic the integration complexity by having integration everywhere. It results in a system that is not consistent, cohesive, and properly constructed. Managers must help the engineers to organize in a way that can contain and manage complexity. A poorly designed system (one that was not designed with quality in mind) will suffer from operational complexity. Many software teams are not responsible for the maintenance of the software they produce, which naturally leads to operational complexity. When the person who built the system is responsible for operating the system, there is a natural reduction of operational complexity. Quality assurance is design integrity from the beginning. It empowers everyone in the organization to say there is a quality problem and be rewarded for it. The cheapest time to discover a defect is in the design and requirements phase — before you build anything. In market complexity, it is helpful when people building the software and systems understand the nature and requirements of the people they are serving, and the nature of the complexities that those people as individuals and groups will face. To contain complexity, you must identify capabilities. Understand the purpose and goal of what you are trying to build. Determine the minimum number of capabilities that can be integrated to solve the main problem. Sometimes you identify a way to have fewer capabilities to solve a problem, which can be used as an evaluation and analysis tool to help you improve existing systems. To enforce constraints, you must first find the appropriate constraints for the system. Once you have the constraints, you can validate things against them, like the capabilities. Encapsulation is one of the most important ways to allow a team to be most effective by defining and clarifying what communication can stay inside the team, and what communication is allowed outside the team for evaluation.