Quantitative Research · KAEL Editorial Department
Data Leakage: Invisible Shortcuts in Model Evaluation
Data leaks expose models to information they should not be aware of during evaluation. It may significantly improve experimental scores, but cannot be reproduced in new environments.
Troubleshooting from the processing flow
If all the data is used to calculate the normalization parameters first, and then the training and evaluation sets are divided, the evaluation information may enter the process in advance. The fitting range of the processor also needs to be consistent with the model training range.
Check sample relationships
Adjacent time segments or repeating entities may be highly similar. Simple random partitioning will cause relevant samples to appear on both sides, making the evaluation task easier than real applications.
Create quarantine rules
Encapsulate data partitioning, feature calculation and evaluation into fixed processes. Any added features indicate when they will be available and how they were generated, allowing review to occur prior to experimentation.
