Choosing the Right Metrics

Author: Joe H., Inflow Engineer

When we started this series, I stated that engineering is the art and science of taking extremely complex systems and getting them to do approximately what you want them to do. In my previous post, Defining a Problem, we discussed how important it is to properly define exactly what it is you want a system to do and we also discussed metrics. In this post, we’re going to dig deeper into just what a metric is and how to use them to measure success.

A metric is a measurement or set of measurements used to determine success. For example, math tests are used to determine a student’s mastery of mathematical concepts. Typically, success is determined by one metric, the number of correctly solved problems, completed with the time limit of the test divided by the total number of problems on the test. If you get a certain number of problems correct, you’ve succeeded in your task. This seems obvious, but take a moment and reflect on what other metrics could possibly be used to gauge mathematical aptitude. For example, we could measure the amount of time spent on each question or the amount of time spent studying for the test. Would these be good indicators of mathematical skill?

These are the types of questions any engineer has to ask when approaching a design problem. In the case of a math test, we need to figure out if someone can solve math problems correctly, which is something we can measure completely with our first metric (number of problems solved). If we add a second metric, time spent on each problem, we might get a better idea of what types of math problems are easiest for the student, but we don’t actually gain any more insight into how likely they are to get a correct answer. Likewise, measuring the time spent studying isn’t going to suddenly change our understanding of what the student knows. If we’re viewing the student as a black box which solves math problems, study time and time spent per question are not meaningful measurements of success.

So, what happens if we change the purpose of our test from an analytical tool measuring how well a student solves problems to a diagnostic tool measuring how well a student is learning math? Suddenly, our additional metrics make a great deal of sense. We want the additional insights in order to make changes that will allow the student to become more proficient at solving math problems. We’re no longer analyzing the student’s mathematical ability, we’re trying to determine how to improve the student’s mathematical ability. We might also want to add a metric measuring the increase in performance over time, to make sure that we’re getting the results we want. The metrics we choose vary based on what we’re trying to accomplish.

This type of detailed understanding of what problem you’re trying to solve is the key to picking good metrics. It’s not enough to know what you’re trying to measure, you need to know why you’re measuring it. If an engineer is designing a bridge and I want to make sure that the bridge will stay standing, I’m going to use metrics that relate to the amount of stress the bridge can withstand, how well the materials hold up to environmental conditions, etc. All of my metrics will relate to the bridge and how it performs. If instead I want to measure the engineer, I’m going to use a different set of metrics. I’ll look at the amount of time taken to design the bridge, how many revisions of the design are needed, if the engineer followed good design practices, and maybe some other metrics related to engineering processes. In this case, my metrics are related to the act of designing the bridge instead of the design of the bridge.

As you can see, generating proper metrics isn’t as simple as it might appear, but learning how to clearly measure the success of a project or task can lead to a massive reduction of time and effort spent to achieve good results. There’s a reason that the terms “metrics-based management” and “performance metrics” have become buzzwords in many industries. Good metrics reduce the amount of time spent solving problems that aren’t relevant to your project. Perhaps even more importantly they allow for an increased amount of creative problem solving. Looking back to our first example, where we viewed the student as a problem solving black box, we can see that by focusing on the results instead of the process allows for all sorts of flexibility as far as the student’s approach to studying. We’ve enabled out of the box thinking, while still ensuring that the solution we reach meets our needs.

Every project and problem requires its own set of metrics, but the basic approach to developing them is fundamentally the same across the board. Again, this type of thought process is something that you can implement in your profession right away. You don't need new equipment or special training, you just need to take the time to look over your normal tasks and projects, and ask yourself what you're doing, why, and how you're measuring success.

-Joe H., Inflow Engineer

 

At Inflow we solve complex terror and criminal issues for the United States Government and their partners, by providing high quality and innovative solutions at the right price through the cultivation of a corporate culture dedicated to being #1 in employee and customer engagement. We Make it Matter, by putting people first! If you are interested in working for Inflow or partnering with us on future projects, contact us here