#58 - What pain-driven discovery misses

The problem with pain

For decades, the dominant method for uncovering product & design opportunities has been to find pain.. You talk to users, observe their struggles, and build solutions to what they report as broken or frustrating. This method is well-developed and widely taught. It is also incomplete in a specific and important way.

Pain-based discovery is built on the assumption that the problems worth solving are the ones people feel and can report. But some of the most significant design failures are ones that nobody reports. It's less because the pain points are minor, and more because the people affected have stopped experiencing them as pain at all. In other words, they have adapted. The problem has been absorbed into normal practice.

In this blog post, I argue that a distinct category of design problem exists alongside felt pain, one that requires different methods to detect and different reasoning to act on. Understanding this category has some practical consequences for how products are built and why some product improvements produce incremental gains while others redefine the category entirely

Misfits and context

Long-time readers of this blog know the name I'm about to throw in. I've mentioned it at least five times already. Yes, it's that architect - Christopher Alexander. In his 1964 book Notes on the Synthesis of Form, he introduced a precise vocabulary for design problems. He defined a misfit as a failure in the relationship between a designed form and the context it inhabits. A form misfits its context when it makes demands the context cannot meet, or when the context makes demands the form cannot satisfy.

This definition is deliberately objective. The misfit exists in the structural relationship between form and context, not in anyone's subjective experience of using the form. A building that handles wind poorly has a misfit with its climate whether or not the inhabitants feel cold. They may have simply learned to avoid certain rooms in winter, or purchase machines that generate heat to counteract the effect of this misfit.

This definition of misfit is important to distinguish between two different types of design. Alexander contrasted two kinds of design cultures.

• In what he called unselfconscious cultures, such as traditional vernacular building traditions, forms evolved slowly through many iterations of small corrections. When a form failed its context, someone noticed and adjusted. Over generations, accumulated corrections produced forms that fit their contexts well. No single designer held the complete solution in mind, but the system as a whole was running continuous, distributed misfit correction.
• In selfconscious design cultures, a single designer or team must hold the entire problem in mind at once. Alexander observed that this produces a characteristic failure: the variables that can be clearly articulated receive attention, while those that are harder to express get underweighted or ignored. The map, as the philosopher Korzybski put it, is not the territory. The designer's model of the problem is not the problem.

Alexander later developed a method for decomposing design problems into sets of misfit variables and grouping them by inseparability. Misfit variables that are tightly linked, where a change that fixes one affects another, should be treated together. This clustering logic anticipates ideas that software engineering would later develop independently, under different names ("high cohesion, loose coupling" principle). "Notes on the Synthesis of Form" is a very compelling book, because it describes a very rational approach to design.

However, Alexander subsequently abandoned the formal decomposition method proposed in this book shortly thereafter. In a follow-up essay, A City is Not a Tree, he argued that designed structures tend toward oversimplified hierarchies while natural structures involve overlapping, semi-lattice relationships that decomposition cannot capture. In other words, decomposition produces a map that's too clean and neat to accurately depict reality. The formal decomposition method was a scaffold for developing intuition about the design space, not a replacement for it. Once he gained the intuition, he abandoned the method.

What survived from Alexander's work, and what is most relevant here, is the core distinction between misfits and felt experience. A misfit can exist without producing pain. This is the premise on which everything that follows depends.

Why misfits get absorbed

A misfit becomes absorbed when the people living with it have compensated around it so successfully that it no longer generates a felt complaint. The compensation is real and often ingenious. But it is a workaround, not a solution. The structural failure in the form-context relationship persists even as its effects become invisible.

The compensation has several consequences worth exploring:

• First, it carries a cost that gets misattributed or becomes invisible. The team that exports data to a spreadsheet every week to do analysis the product should support natively is spending real time and attention, but they have stopped experiencing it as a problem because they have built the workaround into their workflow.
• Second, the compensation lowers expectations. Users who have adapted to a structural failure develop a diminished model of what a product can do. They stop pushing for something the product cannot provide because they have found an alternative path.
• Third, the absorbed misfit suppresses the demand signal that would otherwise drive improvement. In its absence, attention flows to more visible, more recently painful problems.

This is why pain-based discovery methods, however well-executed, have a systematic blind spot. They are tools calibrated to detect phenomenological signal. A bit mouthful, but it simply means that they find what people feel and can articulate. They do not find structural failures that have been successfully compensated around, because the phenomenological signal has been suppressed by the very adaptation that makes users competent. Some folks even build entire professions around compensating for such absorbed misfits.

It would be wrong to say that absorbed misfits produce no signal at all. A more accurate description is that they produce weak, misattributed, or behaviorally-expressed signal that is easy to miss or misread. The signal appears in workarounds that users describe without complaint, in behavioral patterns that diverge from stated preferences, in edge users whose circumstances strip away the compensating behavior, and in clusters of small frustrations that share a structural cause nobody has named.

Evidence from historical cases

What are some examples in which uncovering absorbed misfits led to building category-defining products? In this section, we'll look at three examples including physical and digital products.

The Spreadsheet

The clearest documented case of an absorbed structural misfit producing a transformative product outcome is VisiCalc, released in 1979.

Before electronic spreadsheets, accounting and financial analysis were performed on paper grids. Manual recalculation after any change to an input was simply how the work was done. It was not experienced as a failure. It was professional practice, and expertise at it conferred genuine status.

Dan Bricklin, the co-creator of VisiCalc, later recalled the reaction differential when he demonstrated the software to different audiences.

"In those days, if you showed it to a programmer, he'd say 'Yeah, that's neat. Of course computers can do that - so what?' But if you showed it to a person who had to do financial work with real spreadsheets, he'd start shaking and say, 'I spent all week doing that.' Then he'd shove his charge cards in your face." - Accounting AI needs its VisiCalc moment

The accountant was responding to the sudden revelation that they had been spending days on something that could take minutes. The programmer had no such revelation because they had no compensating behavior to have resolved.

This reaction differential is the empirical signature of an absorbed structural misfit. The people whose expertise was organized around the compensating behavior had the visceral response. The people who stood outside the practice saw nothing remarkable. In other words, the programmer understood what the software did but evaluated it against what computers are capable of, and found nothing surprising. The accountant evaluated it against what their own work cost them, and found everything surprising

The labor market consequences confirmed the structural nature of the misfit. In the decade after VisiCalc's release, the number of accounting clerks in the United States fell by hundreds of thousands, while the number of accountants and auditors grew substantially (not just because of VisiCalc, but it surely had some effects). The misfit had not been about making existing accounting tasks faster. It had been suppressing an entire category of analytical work that became possible once the constraint was removed.

Container Shipping

The second well-documented case involves the shipping industry. Before the introduction of standardized containers in 1956, cargo was loaded and unloaded piece by piece, by hand, exactly as it had been for centuries. A single shipment from Brooklyn to Germany could involve nearly two hundred thousand separate items handled individually. Loading a ship cost around six dollars per ton and took days.

Malcolm McLean, a trucking company operator, watched this process for the first time in 1937 while waiting hours at the Hoboken docks for his cargo to be unloaded. He saw it as waste. Dockworkers did not see it as waste.

"I had to wait most of the day to deliver the bales, sitting there in my truck, watching stevedores load other cargo. It struck me that I was looking at a lot of wasted time and money." - McLean's account in The Box

It was the industry. The International Longshoremen's Association had built an entire labor structure, wage negotiation history, and professional identity around the practice.

When McLean launched the first container ship voyage in 1956, the head of the longshoremen's union was asked what he thought of the new vessel. He replied that he would like to sink it. This was not irrational. His members' livelihoods were organized around the compensating behavior McLean had just rendered unnecessary. The cost of containerization to the shipping industry was sixteen cents per ton, a reduction of roughly thirty-six times. That gap had existed, invisible, for decades before anyone resolved it.

When McLean later expanded to transatlantic routes, arriving in Rotterdam four weeks faster than previous ships when loading time was included, Dutch shipping executives booed his team's celebration. The expert resistance was again organized and explicit, and again it came from the people whose expertise was built on the absorbed misfit.

The iPhone's Interface

The iPhone case is more complex, and the complexity is instructive. There were not one but several misfits in the smartphone market of 2007, and they operated at different levels with different phenomenological profiles.

The felt pain was real and well-documented. BlackBerry users loved their physical keyboards and said so. RIM executives correctly identified that the iPhone's software keyboard was a genuine degradation in typing speed. Their criticism was technically accurate. The keyboard was a felt preference, not an absorbed misfit.

But beneath the keyboard debate sat a structural misfit that nobody in the industry was articulating. In his 2007 keynote, Steve Jobs named it directly.

"They all have these keyboards that are there whether or not you need them to be there. And they all have these control buttons that are fixed in plastic and are the same for every application. Well, every application wants a slightly different user interface, a slightly optimized set of buttons, just for it. And what happens if you think of a great idea six months from now? You can't run around and add a button to these things. They're already shipped." - Steve Jobs, Macworld keynote, January 9, 2007.

You can watch the full keynote here (or read Commoncog's case study):

Every smartphone had a fixed physical interface, with buttons molded in plastic that were the same for every application and could not change after the device shipped. Every application needed a slightly different interface. Software that was not yet written could not add a button to a device already in users' hands. The form was permanently mismatched with a context where software applications have radically different and evolving interaction requirements.

BlackBerry users were not complaining about this. They had never experienced a phone where the interface could adapt to the application. They had no basis for the complaint. Their compensating behavior, accepting a single fixed interface for all applications, was so complete that the misfit was structurally invisible even to the most sophisticated observers in the industry.

RIM's response to the iPhone confirmed the misdiagnosis. The BlackBerry Storm attempted to replicate physical keyboard feel through a clickable touchscreen. It solved the wrong problem. The felt pain layer received a solution. The structural misfit layer was not touched. The product failed.

The App Store, launched a year after the iPhone, revealed in retrospect how severe the fixed-interface misfit had been. The variety and specialization of software interaction that became possible once interfaces could be software-defined was so large that the prior constraint became retrospectively obvious. This is characteristic of absorbed structural misfit resolution: the magnitude of what becomes possible is only legible after the constraint is removed.

Is this just another name for pain?

As I wrote this blog post, I felt an objection arises naturally: is an absorbed misfit simply a pain point that is hard to find? Is this a distinction in kind or just in degree of signal strength?

The distinction is in kind, and it matters for several reasons.

Pain and structural misfit differ in their epistemological status. Pain is phenomenological: it exists in a person's experience and requires a subject to feel it. A structural misfit is relational and objective: it exists in the relationship between a form and its context, independent of whether anyone experiences it. A building that misfits its climate has a real structural failure whether or not the inhabitants have adapted. The failure does not disappear when the compensation succeeds. It persists, generating costs that are misattributed or made invisible by the adaptation.

They also differ in what their resolution produces. Resolving pain produces relief. The product becomes better at the thing it was already doing. Resolving an absorbed structural misfit tends to produce category redefinition. Users do not merely feel better served. They discover a new model of what the product can do, because the compensating behavior that had constrained their expectations is no longer necessary. The accountant who spent a week on what VisiCalc does in minutes does not just save time. They develop a new understanding of what financial modeling can be. This retrospective revelation, the sense that the old way was not merely slower but wrong, is the phenomenological signature of absorbed misfit resolution and does not occur with pain relief alone.

The practical consequence is that they require different instruments to detect. Pain-based discovery finds what people feel and can articulate. It is well-suited to identifying the former and systematically blind to the latter. Detecting absorbed misfits requires examining the structural relationship between form and context directly, looking for compensating behaviors that users describe without complaint, identifying workarounds that have become normalized practice, and seeking out people whose circumstances have stripped away the adaptation.

Expertise and resistance

Two distinct patterns of expert reaction appear across the cases above, and distinguishing them matters for how resistance should be interpreted in practice.

In the VisiCalc case, the domain experts - accountants whose professional practice was organized around manual recalculation - had the most visceral positive response to the demonstration. They recognized the resolution immediately because they were the ones absorbing the cost. Programmers, who stood outside the domain, saw nothing remarkable. Their indifference was not resistance, but the absence of any skin in the game. The misfit was only visible from inside the practice it had deformed.

The containerization and BlackBerry cases follow a different pattern. The longshoremen and RIM executives were not absorbing a cost imposed on them by the misfit. Their competence, livelihood, and professional identity were organized around the compensating behavior itself. Loading cargo piece by piece was skilled work that conferred status and supported an entire labor structure. Typing on a physical keyboard was a genuine capability that RIM had engineered to a high level.

When the misfit was resolved, what disappeared was not a burden but a foundation. The longshoreman who wanted to sink McLean's ship was correctly perceiving that his livelihood had been made unnecessary. The RIM executives who identified the iPhone's software keyboard as inferior were assessing the present accurately while blind to the structural constraint the iPhone had resolved, because that constraint was the ground their expertise stood on.

These two patterns describe different relationships between expertise and compensating behavior:

• Experts who have been absorbing a misfit tend to recognize its resolution with unusual force.
• Experts whose competence is organized around the compensating behavior tend to resist resolution, and their resistance is technically coherent at the level at which they are operating.

The first confirms that a genuine absorbed misfit existed and that the resolution addresses it. The second indicates that the resolution threatens a form of expertise that depended on the problem remaining unsolved.

The implication is, I'm afraid, not super duper practical, or at least it's complicated. Expert resistance is not uniformly informative or uniformly dismissible. When resistance comes from people whose competence is built on the compensating behavior, it is likely to be high and technically grounded, and it is precisely at this point that it is least informative about whether the resolution is correct. The question to ask is not whether experts are resisting, but what their expertise is built on. If the answer is the compensating behavior itself, their resistance is evidence of absorption, not evidence against the proposed change.

How to detect absorbed misfits in practice

If absorbed structural misfits produce weak or misattributed signal rather than no signal, then does it mean we can just stop doing user research? God no. We just need to sharpen our toolkits.

Several approaches are more likely to surface absorbed misfit signal than standard pain-based discovery. Looking for workarounds that users describe as normal practice rather than complaints is one. When someone describes their process and mentions, without frustration, that they export data to a spreadsheet every week or send files to themselves by email, they are describing a compensating behavior. The absence of frustration is worth examining, not accepting at face value. This is why it's so valuable to actually sit and watch someone work, because you can get a lot more context from behaviors as opposed to relying on their ability to articulate and explain problems.

Seeking out edge users whose circumstances have prevented them from developing compensating behavior is another. New employees who have not yet learned the workaround, users migrating from a different system, people under time pressure who cannot afford to compensate, all of these populations experience the underlying misfit more directly than established users whose adaptation is fully automatic. For example, asking new employees to participate in usability testing in which they first sign up and use your product is particularly helpful.

Noticing when clusters of small complaints share a structural cause that none of them name is a third approach. A misfit that manifests across many surface frustrations, none of which seem individually worth prioritizing, may point at a single structural failure that, if resolved, would address all of them simultaneously.

Another approach is to really use the product a lot and develop intuitions around what sort of misfits are absorbed into your workflows. This is a bit more tricky, because it requires you to first invest a lot of time using your own product, and to be able to step back from your own compensating behaviors.

Treating the absence of complaint in an area as potentially informative rather than as evidence of satisfaction is the most counterintuitive but perhaps most important perspective shift. When users seem to have no opinion about some aspect of a product, it is worth asking whether the absence of opinion reflects genuine contentment or reflects an adaptation so complete that the question no longer occurs to them.

The limitation of structural misfit analysis

Christopher Alexander's formal decomposition method was eventually abandoned because it could not handle the semi-lattice structure of real design problems, where variables cluster in overlapping and non-hierarchical ways. The method was useful as a scaffold for thinking but insufficient as a complete account of design.

The same honest limitation applies here. Structural misfit analysis is a diagnostic frame, not a complete design methodology. It tells you where to look and what kind of problem you might be dealing with. It does not tell you what the good form would be, only what the current form is failing to be. Alexander's own subsequent move toward pattern languages, catalogs of positive exemplars rather than negative constraints, reflects this limitation.

There is also a genuine risk of false positives. It'd be dishonest of me not to look for counterfactuals. Not every workaround reflects an absorbed structural misfit worth resolving. Some workarounds are load-bearing: users have not just compensated around a misfit but have reorganized their practice in ways that depend on the misfit remaining.

Google Wave correctly diagnosed that email and documents were structurally separate in ways that created friction. The diagnosis was accurate. But the workaround, keeping communication and documents separate, had become so deeply embedded in how people organized their cognitive and social workflows that resolving the misfit disoriented users rather than liberating them. The product failed not because the structural analysis was wrong but because the compensating behavior had become the practice, and removing the misfit required unlearning the adaptation. Also the UX was really horrible 😄

The practical implication is that identifying an absorbed structural misfit is a necessary but not sufficient condition for deciding to resolve it. It is also necessary to assess whether the compensating behavior has become load-bearing, whether users have reorganized their practice around the workaround in ways that will resist the resolution, and whether the context is stable enough that resolving the misfit will remain relevant after the resolution is complete.

Conclusion

Design and product practice has well-developed frameworks for finding what people feel and can articulate. These frameworks are genuinely useful and should not be abandoned. But they have a structural blind spot: they cannot find what has been successfully compensated around.

The concept of the absorbed structural misfit is useful for identifying this blind spot. A misfit exists in the relationship between a form and its context, independent of whether anyone experiences it. When users adapt successfully to a misfit, the phenomenological signal disappears. The structural failure persists. The costs of compensation persist, invisible and misattributed. The suppression of demand that the misfit creates persists, limiting what users expect and what products become.

Finding absorbed misfits requires different methods than finding pain: looking for normalized workarounds rather than complaints, seeking edge users who have not completed the adaptation, examining structural relationships between form and context directly rather than relying solely on phenomenological report. It also requires a tolerance for the particular kind of expert resistance that absorbed misfits generate, and the analytical discipline to distinguish that resistance from the more ordinary kind that reflects accurate evaluation.

The payoff is that it sometimes helps you to explain a category of design problem that's under-discovered because it is hard to find. The gap between what a form demands of its context and what the context can actually provide does not close when users stop noticing it, it simply waits.