Delayed freshness feedback misaligns relay scheduling decisions

The paper introduces a closed‑loop reinforcement‑learning framework for managing bundles at a deep‑space relay, and it immediately raises the question of how a relay can make effective scheduling choices when the only performance signal—excess age of the information on Earth—arrives long after a transmission decision.

In the described network, intermittent contacts, long propagation delays, and limited storage force the relay to decide which incoming bundles to forward and which to drop. The relay’s objective is to minimize a goal‑oriented loss that penalizes excess age, defined as \(\Delta_i^{e}(t)=\bigl[\Delta_i(t)-d_i(t)\bigr]^+\), where \(\Delta_i(t)\) is the Age of Information for source \(i\) and \(d_i(t)\) is a source‑dependent delay baseline. Because the excess‑age term is evaluated on Earth, the relay does not observe its value at the moment it chooses an action; it only learns the consequence after the bundle has traversed the long propagation path and after the contact window has closed.

This temporal separation creates a feedback‑delay coupling: the control policy must be updated based on a reward that is temporally displaced from the action that generated it. The learning algorithm therefore faces a credit‑assignment problem; it must infer how a particular transmission (or omission) contributed to the excess‑age observed later, while contending with stochastic arrivals of new bundles and uncertain future contacts. The paper addresses the problem by formulating relay control as a constrained semi‑Markov decision process (CSMDP) and solving it with masked Lagrangian Proximal Policy Optimization (PPO). Feasibility‑aware action masking restricts the policy to actions that can be completed within the remaining contact time, ensuring that only physically possible transmissions are considered.

The learning system must therefore develop an internal model of how buffer evolution and future contact opportunities will affect the excess‑age metric. The paper’s simulation results, obtained with DTNSim, show that the proposed approach improves age‑based utility and inter‑flow fairness relative to baseline schedulers.