Let’s talk about black boxing. It’s a fundamental skill for anyone tackling large and complex subjects, but it’s rarely mentioned as something that can be actively trained.
What is black boxing? It’s the act of separating the what from the how. It’s the intention to be satisfied with a systems interface while remaining willfully ignorant of its implementation details.
In the land of software, this is a constant balancing act. There will always be new frameworks, languages, libraries that you can dig into. When we use a library or framework, we are choosing to ignore how it works under the hood, because it’s not our job to care. For instance, when we use os.Open, we don’t have to read its implementation to understand that it’s opening the file for reading or writing. This only works when the interface is well-designed. John Ousterhout captures it well: “A deep module is a good abstraction because only a small fraction of its internal complexity is visible to its users”.
Knowing when to explore further or when to remain satisfied with a black box is a skill. Just like any other skill, it needs to be flexed frequently enough to grow your intuition for when it is necessary to go beyond. In my opinion, less experienced engineers should do this more often, even when it seems unnecessary. Follow your curiosity. Experienced engineers have the advantage of exposure to more systems. They do not need to probe as often because they’ve seen a multitude of similar systems in the past and can reason about how the new one might be implemented. That said, regardless of your experience, it’s paramount that you keep tabs on these boundaries and actively disambiguate them over time. It’s the only way that you’ll be able to build your intuition and learn how complex systems work at a deeper level.
Exploration is a transaction. You are trading some amount of your time for the possibility of insight. I’m saying “possibility” here because there’s no guarantee that you will get the answers you’re looking for. Keep this in mind and really consider if the black box you face is worth investing time into or if you can continue your task treating it as a black box. Black boxes can be thought of as vertices with no outbound edges. They may have many incoming edges, but the node itself has little information. Black boxes are the areas of fog in your mental map. It’s important to keep in mind that black boxes aren’t binary. Uncovering details is a gradient. The more you discover, the more fog of war you alleviate.
So when is it worth the investment to dig deeper? Going back to the graph theory analogy, the higher the in-degree of a vertex, the more valuable it could be to explore. The in-degree of a vertex is just the number of edges pointing into that vertex. In this context, a high in-degree means that this particular node has a high spatial or temporal affinity to its neighbors. Taking the time to understand this particular node could prove useful because it’ll strengthen the connection to its neighbors.
Important
Building a mental map can be difficult to do. Luckily there are tools like Obsidian, Notion, or personal wikis that are great for this.
Tracking these black boxes is a straight forward process. Whenever you’re learning a new topic and you hear a term that you reckon is important enough that you don’t fully grasp, jot it down. Just what you know at the current moment, don’t go any deeper than that! If you run into the same term again in a different context, add an edge to that vertex. An edge signifies a connection of some sort. Once the vertex reaches a certain in-degree value (this can be an adhoc decision), then it’s probably a good candidate for exploration.
Let’s take a concrete example. You are working on a backend service and your API uses HTTP. When you test your code in integration suites, you make real connections to your service from a test harness. Perhaps you’ve also heard that there’s different versions of HTTP that offer possible optimizations for your service. Given this, HTTP has a high affinity with the work you do making it a great candidate to explore. You could dive into the differences between HTTP/1.1, HTTP/2, and HTTP/3, or look at how data is structured at the protocol level.
You should try to have a goal in mind when digging into something. Targeted exploration helps you stay on track and helps avoid getting lost. Keep your target in mind as you go and adjust if you notice yourself going off the rails.
Aside
Debugging an issue originating from your framework is a great excuse to do a deep dive! Most libraries and frameworks are open source. Since you’ve got a specific goal in mind (solve the problem), it’s easier to explore the codebase and narrow your scope.
The alternative approach is sandbox exploration. It’s similar to when you play a video game and choose to ignore the main quest. I actually do this quite a bit when I’m curious or feeling inspired. If you’re short on time and have a busy schedule, then this might not be realistic for you.
Whatever the case may be, stay curious and continue to expand your mental map. I hope this gives you some concrete tips on when it’s the right time to explore versus when you should continue relying on those abstractions.