Recursive Approximation Amplification in Constrained Computation

The recent proposal of a Homomorphic Advantage Operator (HAO) to curb the “Bellman drift” that appears when deep reinforcement‑learning (RL) agents are run under fully homomorphic encryption (FHE) is not an isolated engineering tweak. It exemplifies a recurring structural flaw: when a recursive algorithm is transplanted onto a substrate that can only supply approximate evaluations of its primitive operations, the approximation error is fed back into the recursion, and the error grows unchecked until the computation collapses.

In the RL setting the Bellman equation defines the value of a state as a function of the values of successor states. The standard update relies on non‑linear operators such as max, softmax, and activation functions that are evaluated exactly in plaintext. Under FHE the same operators must be expressed as low‑degree polynomials because only addition and multiplication are homomorphically tractable. Each polynomial replaces a true non‑linear mapping with a bounded approximation. The HAO paper notes that “replacing non‑linear operations with polynomial approximations … diverge catastrophically due to a unique recursive error phenomenon known as the Bellman drift.” The drift is not a statistical anomaly; it is the deterministic consequence of feeding an approximated value back into the same Bellman recursion, where each iteration compounds the prior deviation. The resulting value sequence no longer converges to the true fixed point but spirals toward the limits of the polynomial’s representational range.

The same mechanism has been observed whenever a fixed‑point iteration is forced onto a numerically limited platform. In 1952 John von Neumann published a seminal analysis of round‑off error in digital computers, demonstrating that even linear iterative solvers can become unstable when each arithmetic step is rounded to a finite number of bits. The analysis showed that the error introduced at each step is multiplied by the system matrix, and if the matrix’s spectral radius exceeds one, the error grows geometrically. The situation is mathematically identical to the Bellman drift: a recursive update, an approximation at each step, and a multiplicative amplification that overwhelms the intended signal.

A historical analogue in control engineering appears in the 1960s when digital PID controllers were first deployed in aerospace guidance. Early implementations used 12‑bit fixed‑point arithmetic for integrator and differentiator calculations. Engineers observed that the control loop would oscillate wildly even though the underlying continuous‑time design was stable. The cause was traced to quantization error in the integrator term, which, when summed over many cycles, produced a drift that fed back into the error signal. The remedy was to redesign the controller with dithering and higher‑precision arithmetic—essentially a hardware analogue of the HAO’s software stabilization.

The phenomenon also surfaces in molecular biology. Manfred Eigen’s error‑catastrophe theory, formulated in the 1970s, posits that self‑replicating polymers such as RNA experience a threshold replication fidelity. Below the threshold, each replication introduces mutations that are recursively incorporated into the next generation, leading to an exponential loss of information. The mathematical description is a recursion where the fidelity term is an approximation of the ideal replication process. When the fidelity falls below a critical value, the error term dominates, and the system collapses into a random sequence pool. The “error catastrophe” is a biological incarnation of recursive approximation amplification, with the fidelity of enzymatic copying playing the role of the polynomial approximation in the RL‑FHE context.

Financial markets provide a more recent illustration. In the early 2000s, algorithmic trading systems began using discretized risk models that approximated continuous‑time stochastic differential equations with finite‑difference schemes. When market volatility surged, the discretization error in the risk estimate fed back into position sizing decisions, which in turn amplified the error in subsequent risk calculations. The feedback loop contributed to the “flash crash” of May 2010, where a cascade of algorithmic orders, each based on an increasingly erroneous risk metric, drove prices far from equilibrium before human traders intervened. The crash demonstrates that when a decision‑making loop relies on an approximated model, the model’s error can become the driver of market instability.

Even in the realm of software engineering, the classic “limit cycle” problem in digital signal processing mirrors the Bellman drift. A digital filter implemented with quantized coefficients can exhibit a periodic oscillation that never decays, not because the filter design is inherently unstable but because rounding error in the coefficient multiplication is re‑injected at each sample. The phenomenon was documented in the 1990s by researchers at Texas Instruments, who showed that a second‑order IIR filter with 16‑bit coefficients could generate a limit cycle when the input signal was zero, solely due to the quantization error being fed back through the recursion. The fix involved either increasing coefficient precision or adding dithering noise—strategies that correspond to the HAO’s aim of stabilizing the recursive update under constrained arithmetic.

The common thread across these domains is a three‑part causal chain. First, an algorithmic process requires a non‑linear or otherwise exact primitive operation. Second, the execution environment can supply only an approximate version of that primitive, typically because of a hard constraint (finite bits, homomorphic restrictions, biochemical fidelity, or discretized models). Third, the algorithm’s structure feeds the output of the approximated primitive back into itself, often through a summation or maximization step, so that the approximation error is multiplied on each iteration. The multiplication is not accidental; the recursion’s Jacobian matrix frequently contains eigenvalues larger than one precisely because the algorithm is designed to amplify useful signal differences. When the same matrix multiplies the error term, the error inherits the same amplification factor, leading to exponential divergence.

In the case of homomorphic RL, the Bellman drift emerges because the polynomial approximations of max and activation functions have bounded curvature. The Bellman update computes a target value as a sum of a reward and a discounted future value, then applies a max operator to select the best action. When the max is replaced by a low‑degree polynomial, the slope of the approximation near the decision boundary is reduced. Consequently, the gradient of the value function with respect to the state becomes shallower, and the discount factor—normally less than one—fails to dominate the error term. Each iteration therefore adds a residual that is only partially attenuated, and the residual is amplified by the same discount factor applied repeatedly. The HAO attempts to insert a corrective term that counteracts this residual, but the underlying mechanism remains: an approximation error that is recursively recycled.

A parallel can be drawn to the 1992 loss of the Mars Climate Orbiter, where a unit conversion error caused a navigation algorithm to compute thrust vectors with an offset of 0.001 ft · s⁻¹. The navigation loop repeatedly used the erroneous thrust estimate to update the spacecraft’s trajectory, and each update compounded the positional error. The spacecraft eventually entered an atmosphere at a lower altitude than planned and was lost. The incident is often cited as a “communications failure,” yet the technical root is the same recursive amplification of a systematic approximation error—here a unit conversion—within a closed‑loop guidance algorithm.

The persistence of this mechanism across centuries and sectors suggests that any system that couples a recursive decision process with an approximated primitive is vulnerable. The vulnerability does not arise from malicious intent or poor coding style; it is baked into the mathematical structure of the recursion. When designers substitute an exact operation with an approximation, they must either redesign the recursion to be contractive with respect to the approximation error or introduce an external stabilizer that explicitly removes the residual at each step. The HAO’s proposal to add a stabilizing term is one instance of the latter approach, but the broader lesson is that the recursion itself must be re‑engineered to tolerate the error budget imposed by the execution substrate.

The challenge, then, is to recognize when a recursive algorithm has crossed the threshold where its intrinsic amplification factor exceeds the attenuation provided by discounting, damping, or other contractive forces. In numerical analysis this threshold is captured by the concept of “numerical stability”: an algorithm is stable if a bounded perturbation in the input yields a bounded perturbation in the output, regardless of iteration depth. The Bellman drift demonstrates that many RL algorithms, designed under the assumption of exact arithmetic, are numerically unstable once the arithmetic is replaced by low‑degree polynomial homomorphisms. The same instability was identified in the 1960s by James Wilkinson, who showed that Gaussian elimination without partial pivoting can produce arbitrarily large errors when the matrix is ill‑conditioned. The remedy—pivoting—restructures the computation to avoid feeding large intermediate values back into the elimination steps, a strategy analogous to redesigning the Bellman recursion to avoid feeding polynomial error back into the value estimate.

In practice, engineers have often responded to such instability by increasing the precision of the underlying arithmetic. The early 2000s saw a shift from 32‑bit floating‑point to 64‑bit double‑precision in scientific computing precisely to reduce round‑off accumulation in iterative solvers. However, precision upgrades are not always feasible. FHE, by design, incurs a massive ciphertext expansion and computational overhead that makes high‑degree polynomial approximations prohibitively expensive. The same constraint applies in embedded control systems where power budgets limit word length, and in synthetic biology where enzyme fidelity cannot be arbitrarily improved without metabolic cost. Thus, the only viable path is to alter the algorithmic structure itself.

One alternative that has been explored in the control literature is the use of “dead‑beat” observers, which compute the exact state in a finite number of steps by inverting the system dynamics. When the inversion is performed with approximated arithmetic, the observer can be designed to include a correction term that nullifies the accumulated quantization error at each step. Translating this idea to RL would require a Bellman update that explicitly subtracts an estimate of the polynomial approximation error, perhaps using a secondary network trained to predict the drift. Such a “drift estimator” would be analogous to the HAO’s stabilizing operator, yet it would be learned rather than hand‑crafted, potentially adapting to the specific error landscape of the chosen polynomial basis.

Another avenue lies in reshaping the recursion to be inherently contractive. In the 1990s, researchers introduced “soft‑max” Bellman updates that replace the hard max with a smooth, temperature‑scaled log‑sum‑exp function. The soft‑max is more amenable to polynomial approximation because its curvature can be captured with lower‑degree terms while preserving a bounded Lipschitz constant. By ensuring that the effective gain of the recursion stays below one, the error introduced by the polynomial approximation is guaranteed to decay over iterations. This approach mirrors the design of “contractive autoencoders” in deep learning, where the encoder is constrained to have a Jacobian norm less than one, thereby limiting the propagation of input perturbations.

The persistence of recursive approximation amplification raises a question that remains unresolved: how can one systematically determine the error amplification factor for an arbitrary polynomial approximation within a given recursive algorithm, without resorting to exhaustive simulation? In numerical linear algebra, condition numbers provide a concise measure of sensitivity, but extending this concept to non‑linear, policy‑dependent recursions under homomorphic constraints is an open research problem. Existing analyses of Bellman drift rely on empirical observation of divergence; a formal bound that relates polynomial degree, coefficient quantization, and discount factor to a guaranteed stability region would allow designers to certify RL agents for FHE deployment before any training begins. The absence of such a theory leaves practitioners to guess whether a chosen polynomial basis will remain within the safe region, a gamble that the HAO paper attempts to mitigate with an ad‑hoc stabilizer but does not eliminate.

Thus, the Homomorphic Advantage Operator is a concrete response to a timeless engineering dilemma: the clash between a recursive algorithm’s desire for exactness and a computation platform’s limitation to approximations. The episode is a modern incarnation of a pattern that has forced redesigns in digital control, numerical linear algebra, molecular evolution, financial engineering, and aerospace navigation. Each field eventually learned to either restructure the recursion, augment it with error‑correcting feedback, or accept higher precision. The RL‑FHE community now faces the same choice, and the decision will shape whether privacy‑preserving agents can be deployed at scale or remain confined to simulated environments where exact arithmetic is cheap.

The ultimate test will be whether future homomorphic RL systems can be proven to stay within the stability envelope without auxiliary stabilizers, or whether the very act of preserving privacy will forever demand a layer of corrective logic that anticipates and counteracts the inevitable drift. The answer remains pending, and the underlying mechanism—recursive approximation amplification—continues to loom over any attempt to graft exact‑logic algorithms onto approximate substrates.