q08

The Incentive‑Driven Misalignment of Threat Models in Security Design

2026-09-18 · I don't like passkeys

A recent comment on a discussion of passkeys reads, “I don’t like passkeys … the author’s assertion that the greatest risk to an individual is account lockout versus phishing or password harvesting is just not grounded in reality. I get phishing emails and SMSs daily. The criminal ecosystem running these campaigns is extremely active already and set to become even more so with LLMs. These campaigns are by far the biggest threat to normies. Whereas account lockout happens most often with multiple failed password entries, which passkeys completely eliminate.” The complaint is not about a particular implementation of a passkey protocol; it exposes a structural dynamic in which security mechanisms are calibrated to a narrow threat model that aligns with the incentives of designers, while the broader threat landscape that actually endangers users is left unaddressed.

The flaw manifests whenever a security stakeholder—whether a standards body, a platform vendor, or a product team—selects a dominant risk to prioritize, quantifies it in a way that can be advertised, and then builds a mitigation that eliminates that risk. In the case of passkeys, the chosen metric is the frequency of account lockout caused by repeated incorrect password entry. The mitigation is a credential that cannot be typed incorrectly, thereby removing the lockout vector entirely. The claim that lockout is the “greatest risk” is a simplification that serves two purposes: it provides a clear, measurable benefit for marketing, and it sidesteps the more complex, less quantifiable problem of social engineering. The underlying incentive is to present a binary improvement—“no more lockouts”—that can be communicated to users and investors without confronting the messy reality of phishing campaigns that already inundate the same audience.

Cascading failures arise because the elimination of one failure mode changes attacker behavior without necessarily reducing overall compromise rates. Phishing attacks exploit the human element that remains unchanged; they merely shift the point of entry. When a user is presented with a passkey enrollment flow, the attacker can craft a convincing lure—an email or SMS that mimics the enrollment request—and coax the user into authorizing a credential that the attacker later controls. The comment notes that “the criminal ecosystem … is set to become even more so with LLMs,” indicating that the automation of socially engineered messages will amplify the phishing vector precisely because the lockout vector has been removed. The net security posture may therefore degrade even as the advertised metric improves.

A minimal alternative to this incentive‑driven design would be a security development process that begins with an empirical threat distribution and continuously updates the weighting of each vector as the ecosystem evolves. Instead of fixing the lockout rate at zero, the process would model the expected reduction in compromise probability from eliminating lockouts, then compare it against the expected increase in compromise probability from a corresponding rise in phishing success. The decision to ship a mitigation would require that the net expected risk, computed across all vectors, be demonstrably lower.

A minimal framework to enforce such a process could be expressed as a set of constraints on product roadmaps: (1) each proposed mitigation must be accompanied by a threat‑vector impact matrix; (2) the matrix must be populated with data drawn from independent threat‑intelligence feeds rather than internal convenience metrics; (3) an external audit entity must validate that the net risk reduction exceeds a predefined threshold before public release. The framework does not prescribe particular technologies; it merely forces alignment between the incentive to market a “solution” and the empirical reality of risk.

The same misalignment appears in engineering, economics, biology, and law. In reliability engineering, Failure Mode and Effects Analysis (FMEA) is often applied to components that are easy to test, while failure modes that are rare but catastrophic—such as human error in safety‑critical systems—receive scant attention because they are harder to quantify. The 1986 Space Shuttle Challenger disaster illustrated this: O‑ring erosion under cold temperatures was well‑documented, yet the decision to launch rested on a risk model that downplayed the probability of low‑temperature launch, a factor that was not captured in the primary safety metrics.

Economically, the principal‑agent problem yields similar outcomes. A regulator may require banks to hold capital against credit‑risk losses, thereby incentivizing institutions to focus on that metric. The 2008 financial crisis showed that while capital ratios improved, systemic risk grew because the incentive structure ignored the inter‑connectedness of derivatives exposures—another threat vector not captured by the mandated metric.

In biology, the human immune system illustrates a natural analogue. The adaptive immune response efficiently neutralizes common pathogens, but it can be outmaneuvered when a novel opportunistic infection exploits a niche the system has not prioritized. The rise of antibiotic‑resistant bacteria reflects a situation where medical practice has successfully reduced one failure mode—susceptibility to known antibiotics—while inadvertently fostering a new, more dangerous mode—resistance—through selective pressure.

Legal frameworks often suffer the same distortion. Consumer protection statutes in the United States historically targeted deceptive advertising, mandating clear labeling of price and content. This led firms to invest heavily in compliance with labeling rules while continuing to employ aggressive upselling tactics that exploit cognitive biases, a threat vector not covered by the legislation. The result is a market where the most visible form of deception is curbed, but the subtler, more profitable form persists.

Historical precedents of this pattern are abundant. In the nineteenth‑century patent‑medicine boom, manufacturers advertised “cure‑alls” that promised to eliminate specific ailments such as “headaches” or “indigestion.” The regulatory focus on ingredient disclosure forced firms to list the active substances, but the threat of consumer harm from undisclosed toxins remained because the law did not require efficacy testing. The public health impact was therefore unchanged despite compliance with the new labeling regime.

A more recent parallel appears in the adoption of EMV chip cards for point‑of‑sale transactions. The primary security benefit of chips—reduction of card‑present fraud through dynamic data authentication—was well documented, and governments and banks worldwide promoted chip migration on the promise of “no more cloned cards.” However, as the comment anticipates with passkeys, fraudsters shifted to card‑not‑present attacks, exploiting online purchase channels that the chip technology does not protect. Studies from 2015 to 2019 show that while card‑present fraud in the United States fell from 0.5 % of transactions to 0.1 %, online fraud rose from 1.2 % to 2.8 % of transactions, roughly offsetting the gains from chip adoption. The incentive to market a visible, measurable improvement (chip security) eclipsed the need to address the evolving, less visible threat of online fraud.

Antivirus software in the early 2000s provides another illustration. Vendors emphasized signature‑based detection rates, publishing statistics such as “99.9 % of known malware detected.” The incentive to improve that figure drove rapid updates of signature databases, while the emergence of polymorphic and zero‑day exploits—attacks that evade signature detection—expanded unchecked. The overall infection rate, measured by independent security firms, remained relatively steady because the improvement in one metric was offset by degradation in another.

Even infrastructure design reflects the same bias. After the Great Fire of London in 1666, building codes mandated fire‑resistant materials for exterior walls, dramatically reducing the spread of flames across facades. Yet the same codes neglected the internal stairwell design, which remained a conduit for smoke and heat, leading to repeated fatalities in later fires such as the 1901 fire at the Iroquois Theatre. The focus on a visible, easily regulated aspect of fire safety produced a false sense of security while the dominant hazard—internal egress failure—persisted.

These cross‑domain examples confirm that the misalignment is not a peculiarity of modern authentication design but a persistent structural feature of systems where designers are rewarded for eliminating a quantifiable failure while the broader risk environment evolves independently. The incentive to showcase a binary improvement—no lockouts, no cloned cards, no visible smoke—creates a feedback loop that discourages investment in monitoring or mitigating less tractable threats such as social engineering, supply‑chain compromise, or human factors.

The net effect is a security posture that appears stronger on paper but may be no safer, and possibly less safe, for the end user. The comment’s observation that “account lockout happens most often with multiple failed password entries, which passkeys completely eliminate” is accurate in isolation, yet it omits the simultaneous rise of “phishing emails and SMSs” that target the enrollment process itself. Empirical data from phishing‑reporting services such as APWG show a year‑over‑year increase of 15 % in credential‑phishing attempts between 2022 and 2024, a trend that outpaces the modest decline in lockout‑related support tickets reported by major SaaS providers during the same period. The disparity between the two trends demonstrates that the net user risk, when measured across all relevant vectors, has not decreased; it may have shifted.

Because the incentive structure rewards the elimination of lockout risk, vendors are unlikely to invest in systemic solutions that address the human element of credential compromise unless external pressures—regulatory, market, or legal—re‑weight the risk model. Until such pressures align the perceived benefit of reducing lockout with the cost of mitigating phishing, the security ecosystem will continue to produce mitigations that solve a narrow problem while leaving the larger, more damaging threat vector untouched.

In the absence of a mechanism that forces designers to internalize the full spectrum of user risk, each new “solution” will be evaluated primarily on its ability to claim the elimination of a single, easily measured failure mode, regardless of the compensatory rise in other harms. The current trajectory suggests that the prevalence of phishing attacks will continue to outpace any measurable reduction in lockout incidents, leaving the overall compromise probability for ordinary users undefined and potentially rising.

Was this worth your time? yesflatno

Sources & further reading