Improving Enterprise Data Integrity Through Automated Validation and Schema-Drift Detection in Large-Scale Data Integration Systems

Abstract

Enterprise data integrity has become increasingly important as large-scale integration environments combine heterogeneous source systems, rapidly evolving schemas, complex transformations, and high-volume processing workloads. This study evaluated the effectiveness of automated data validation and schema-drift detection in improving data integrity, detection performance, and operational reliability across enterprise data integration systems. A quantitative quasi-experimental longitudinal design was applied to 128.64 million eligible records obtained from 24 enterprise source systems and processed through 186 integration pipelines, 42 databases, 1,284 tables, 18,736 attributes, 318 schema versions, 2,946 transformation processes, and 38,640 eligible pipeline runs. The analysis incorporated 4,872 validation rules, 1,126 verified schema-change events, and 2,384 recorded data-quality incidents. Following implementation, the overall integrity score increased from 93.1 to 97.4, while invalid-record prevalence decreased from 7.3% to 2.4%, defect leakage from 5.7% to 1.2%, and reconciliation discrepancies from 4.6% to 1.5%. Pipeline success increased from 91.7% to 98.0%, failed runs decreased from 8.3% to 2.0%, reprocessing declined from 14.6% to 4.8%, and service availability increased from 96.2% to 99.1%. Mean detection time decreased from 6.8 to 1.4 hours, while mean resolution time declined from 14.7 to 5.2 hours. Schema-drift detection achieved 97.1% accuracy, 89.5% precision, 95.6% recall, 97.4% specificity, a 92.4% F1-score, and an ROC-AUC of 0.982. All principal pre-post differences were statistically significant at p < 0.001, with Cohen’s d values ranging from 0.93 to 1.41. Ten-fold cross-validation achieved 97.0% accuracy, while independent holdout testing achieved 96.6%, producing a generalization gap of only 0.4 percentage points. The findings demonstrated that automated validation integrated with schema-drift detection substantially strengthened enterprise data integrity, reduced defect propagation and operational failures, accelerated detection and recovery, and maintained robust performance across heterogeneous architectures, data volumes, and schema-complexity conditions.