Built in 2024.
Back to top
In product design, when we release a feature into the wild, we can expect a degree of objectivity in the feedback we get. We can tell if our designs are helping or hurting based on the feedback and metrics after we ship. Getting my hands on these metrics can be challenging at times. My experience has largely been on early-stage products: when I’ve joined a team, our product has reached some level of product-market fit, but my team is at a point where it makes sense to focus on releasing new features instead of optimizing existing ones. Speed tends to get prioritized over quality.
Sometimes metrics aren’t fully defined or they’re not collected in the system. There have been instances where I’ve advocated for my team to slow down and build out product metrics so we can have a better understanding of how our work is performing. Without this type of feedback, there can be no reflection, and without reflection, there is rarely any improvement. I worry that, by moving from one feature to the next and not pausing to reflect, we may be missing important learning opportunities.
Around the post-war era, there was a neighbourhood built in the Netherlands where the electric meters were installed in the basement in some of the houses, and in the front hall of others. The meters were the type where you could see your energy usage at a glance when you walked by them. For the homes with electrical meters in the front hall, when home owners entered their home they could immediately see their electricity usage.
“During the oil embargo and energy crisis of the 1970’s, the Dutch began to pay close attention to their energy use. It was discovered that some of the houses in the subdivision used one third less electricity than the other houses. No one could explain this. All houses were charged the same price for electricity, all contained similar families.” — Thinking in Systems, Donella Meadows.
It turns out that the houses with meters installed in their front halls were using less electricity than their neighbours. A homeowner could see their home’s energy usage every time they walked in, and could then make energy-conscious decisions afterwards. If they noticed that they were using more energy than normal, they could choose to turn more lights off in the house, or watch TV for less time. The electrical meters morphed into a feedback loop. Seeing their energy usage metrics on a regular basis ended up shaping each homeowner’s decision-making processes. There is a powerful lesson in this example. If we want to change a behaviour, one important step is to create a clear and obvious feedback loop.
If I could wave a magic wand, every morning I’d start each day in front of a product metrics dashboard. This might be the equivalent of Mixpanel or Segment. I’d be able to, at a glance, see how users have adopted each new feature that we released, and how it’s impacted the metrics we set for it. I’d know how my work is influencing the company strategy — because I can see things like feature X improved retention by Y% and unlocked Z revenue opportunities. If a feature didn’t perform as expected, after the release I’d have time blocked off in my calendar so that me and my team could reflect on how effective our work was. Understanding what went wrong and adjusting for it could impact all design processes from that point going forward. The results might shine a light on what we neglected to do or look into during the design process, and I could then engineer our future processes to cover off these types of blind spots.
Every product decision is meant to move some sort of metric. The responsibility to define that metric usually falls on a product manager’s shoulders: they are tuned into the top-line company strategy, and in turn they define how their product is going to grow in such a way to push that top-line strategy forward. Product Designers take those metric goals and make them manifest. We can almost think of product development like a freight train steaming forward at a predictable pace. Designs are wooden logs and charcoal fed to the fire of engineering that powers product growth. Engineering needs designs to be ready and on time as their work on the last feature finishes up, to avoid idle time. A designer’s process timeline is often constrained by engineering estimates. The more engineers that a designer supports, the less time they have for process.
It’s difficult to consistently set aside time to reflect on a design process: both individually and as part of a team process. Engineers have retrospectives after each sprint, but designers don’t seem to have an equivalent process ritual. There are reasons for this. On smaller teams, designers tend to be stretched thin. What this translates to is consistent pressure to finish one feature and immediately start another, or, even more difficult, trying to juggle the design for multiple products at a time.I’ve felt like I’ve been on that freight train on the past. Always looking forward, never looking back. Not asking the hard questions like was what we did back then effective? Are we going to repeat the same mistakes? In fact, now that you mention it, did we make any mistakes? Will we know unless a customer yells at us about it?
That customer won’t yell at us until weeks, sometimes months after a design is complete. Prior to entering the industry, I harboured this notion that designers would finish their work, ship it, and the next day they’d hear feedback. Metrics would change, customers would write thank you notes in their little intercom bubbles, everybody would be smiles. Or frowns. Regardless of whether the design was a success or failure, the designer would know right away and be able to process that feedback and learn.
That’s not how it works.
A design is handed off to engineering, and depending on the feature’s complexity, engineering will take weeks, sometimes months, to finish coding the feature. It may take an additional week to be released to production. By the time customers see the feature, that designer has already started on the next feature in their team’s roadmap — and may be deeply engaged in that feature’s design process. It’s a harsh context switch to change out of the current process mindset and back into the former one as feedback starts flowing in.
“When there are long delays in feedback loops, some sort of foresight is essential. To act only when a problem becomes obvious is to miss an important opportunity to solve the problem.” — Thinking in Systems, Donella Meadows.
There needs to be a certain level of thought upfront about documenting a process, setting up the metrics that we want to measure, and reminding ourselves to block time out to compare outcomes against processes. When we’re caught up in execution, improvement becomes an after thought, even though improvement is often, in the long run, a higher leverage activity.
I’ve been trying to think of feedback and reflection as a critical step in my design process. It’s the step that sets up the next feature that I work on to be better than the last, instead of more of the same. Picture a process framework, for example, the double diamond design process. If we visualize the process, it looks like two diamonds side by side; representing four phases of a design process that oscillate between convergence and divergence. Now think of another, fifth step, tacked on to the end. This might represent reflection. If I were to draw my idea of the process, it might look like those two diamonds, side by side, and then a long line connecting the diamonds to a small circle. The line represents the time delay between process and feedback, and the circle is my reminder to block time out to think.
“All my projects now start with a brief: What are my objectives? What do we expect to happen? And they end with a review: What was the outcome? Was my hypothesis correct?” — Note-taking and accountability, Helen Tran.
The end starts at the beginning. Without the intention upfront to be diligent about what success looks like and how we plan to learn from our outcomes, then the reflection may not happen — as seemingly more important things come up the minute we’ve finished designing.