Authors: Augusto Cerqua, Marco Letta, Gabriele Pinto
studying (ML) has gained a central function in economics, the social sciences, and enterprise decision-making. Within the public sector, ML is more and more used for so-called prediction coverage issues: settings the place policymakers intention to determine items most susceptible to a detrimental final result and intervene proactively; for example, focusing on public subsidies, predicting native recessions, or anticipating migration patterns. Within the non-public sector, related predictive duties come up when corporations search to forecast buyer churn, or optimize credit score danger evaluation. In each domains, higher predictions translate into extra environment friendly allocation of assets and simpler interventions.
To realize these objectives, ML algorithms are more and more utilized to panel information, characterised by repeated observations of the identical items over a number of time intervals. Nonetheless, ML fashions weren’t initially designed to be used with panel information, which function distinctive cross-sectional and longitudinal dimensions. When ML is utilized to panel information, there’s a excessive danger of a delicate however major problem: information leakage. This happens when info unavailable at prediction time unintentionally enters the mannequin coaching course of, inflating predictive efficiency. In our paper “On the Mis(Use) of Machine Studying With Panel Information” (Cerqua, Letta, and Pinto, 2025), not too long ago printed within the Oxford Bulletin of Economics and Statistics, we offer the primary systematic evaluation of knowledge leakage in ML with panel information, suggest clear pointers for practitioners, and illustrate the results via an empirical utility with publicly accessible U.S. county information.
The Leakage Drawback
Panel information mix two buildings: a temporal dimension (items noticed throughout time) and a cross-sectional dimension (a number of items, akin to areas or corporations). Commonplace ML apply, splitting the pattern randomly into coaching and testing units, implicitly assumes impartial and identically distributed (i.i.d.) information. This assumption is violated when default ML procedures (akin to a random cut up) are utilized to panel information, creating two most important kinds of leakage:
- Temporal leakage: future info leaks into the mannequin through the coaching part, making forecasts look unrealistically correct. Moreover, previous info can find yourself within the testing set, making ‘forecasts’ retrospective.
- Cross-sectional leakage: the identical or very related items seem in each coaching and testing units, that means the mannequin has already “seen” a lot of the cross-sectional dimension of the information.
Determine 1 exhibits how totally different splitting methods have an effect on the danger of leakage. A random cut up on the unit–time degree (Panel A) is probably the most problematic, because it introduces each temporal and cross-sectional leakage. Alternate options akin to splitting by items (Panel B), by teams (Panel C), or by time (Panel D), mitigate one sort of leakage however not the opposite. In consequence, no technique fully eliminates the issue: the suitable selection depends upon the duty at hand (see beneath), since in some circumstances one type of leakage might not be an actual concern.
Determine 1 | Coaching and testing units underneath totally different splitting guidelines
Two Forms of Prediction Coverage Issues
A key perception of the research is that researchers should clearly outline their prediction aim ex-ante. We distinguish two broad lessons of prediction coverage issues:
1. Cross-sectional prediction: The duty is to map outcomes throughout items in the identical interval. For instance, imputing lacking information on GDP per capita throughout areas when just some areas have dependable measurements. The most effective cut up right here is on the unit degree: totally different items are assigned to coaching and testing units, whereas all time intervals are saved. This eliminates cross-sectional leakage, though temporal leakage stays. However since forecasting is just not the aim, this isn’t an actual subject.
2. Sequential forecasting: The aim is to foretell future outcomes based mostly on historic information—for instance, predicting county-level revenue declines one 12 months forward to set off early interventions. Right here, the right cut up is by time: earlier intervals for coaching, later intervals for testing. This avoids temporal leakage however not cross-sectional leakage, which isn’t an actual concern because the identical items are being forecasted throughout time.
The flawed method in each circumstances is the random cut up by unit-time (Panel A of Determine 1), which contaminates outcomes with each kinds of leakage and produces misleadingly excessive efficiency metrics.
Sensible Pointers
To assist practitioners, we summarize a set of do’s and don’ts for making use of ML to panel information:
- Select the pattern cut up based mostly on the analysis query: unit-based for cross-sectional issues, time-based for forecasting.
- Temporal leakage can happen not solely via observations, but additionally via predictors. For forecasting, solely use lagged or time-invariant predictors. Utilizing contemporaneous variables (e.g., utilizing unemployment in 2014 to foretell revenue in 2014) is conceptually flawed and creates temporal information leakage.
- Adapt cross-validation to panel information. Random k-fold CV present in most ready-to-use software program packages is inappropriate, because it mixes future and previous info. As a substitute, use rolling or increasing home windows for forecasting, or stratified CV by items/teams for cross-sectional prediction.
- Make sure that out-of-sample efficiency is examined on actually unseen information, not on information already encountered throughout coaching.
Empirical Software
For instance these points, we analyze a balanced panel of three,058 U.S. counties from 2000 to 2019, focusing completely on sequential forecasting. We think about two duties: a regression downside—forecasting per capita revenue—and a classification downside—forecasting whether or not revenue will decline within the subsequent 12 months.
We run lots of of fashions, various cut up methods, use of contemporaneous predictors, inclusion of lagged outcomes, and algorithms (Random Forest, XGBoost, Logit, and OLS). This complete design permits us to quantify how leakage inflates efficiency. Determine 2 beneath stories our most important findings.
Panel A of Determine 2 exhibits forecasting efficiency for classification duties. Random splits yield very excessive accuracy, however that is illusory: the mannequin has already seen related information throughout coaching.
Panel B exhibits forecasting efficiency for regression duties. As soon as once more, random splits make fashions look much better than they are surely, whereas right time-based splits present a lot decrease, but life like, accuracy.
Determine 2 | Temporal leakage within the forecasting downside
Panel A – Classification job
Panel B – Regression job
Within the paper, we additionally present that the overestimation of mannequin accuracy turns into considerably extra pronounced throughout years marked by distribution shifts and structural breaks, such because the Nice Recession, making the outcomes notably deceptive for coverage functions.
Why It Issues
Information leakage is greater than a technical pitfall; it has real-world penalties. In coverage functions, a mannequin that appears extremely correct throughout validation could collapse as soon as deployed, resulting in misallocated assets, missed crises, or misguided focusing on. In enterprise settings, the identical subject can translate into poor funding choices, inefficient buyer focusing on, or false confidence in danger assessments. The hazard is very acute when machine studying fashions are meant to function early-warning methods, the place misplaced belief in inflated efficiency may end up in pricey failures.
Against this, correctly designed fashions, even when much less correct on paper, present sincere and dependable predictions that may meaningfully inform decision-making.
Takeaway
ML has the potential to rework decision-making in each coverage and enterprise, however provided that utilized appropriately. Panel information supply wealthy alternatives, but are particularly weak to information leakage. To generate dependable insights, practitioners ought to align their ML workflow with the prediction goal, account for each temporal and cross-sectional buildings, and use validation methods that stop overoptimistic assessments and an phantasm of excessive accuracy. When these rules are adopted, fashions keep away from the lure of inflated efficiency and as an alternative present steering that genuinely helps policymakers allocate assets and companies make sound strategic decisions. Given the speedy adoption of ML with panel information in each private and non-private domains, addressing these pitfalls is now a urgent precedence for utilized analysis.
References
A. Cerqua, M. Letta, and G. Pinto, “On the (Mis)Use of Machine Studying With Panel Information”, Oxford Bulletin of Economics and Statistics (2025): 1–13, https://doi.org/10.1111/obes.70019.
