Share
Export Citation
In-ETL Architecture with Isolation Forest-Based Data Triage for Improving Open Government Data Quality
Asdar M.
2026 IEEE International Conference on Artificial Intelligence and Mechatronics Systems Aims 2026
Abstract
Implementing Open Government Data (OGD) in Indonesia faces data quality challenges due to inconsistencies and anomalies across Regional Apparatus Organizations. This study proposes a proactive approach by integrating multi-stage rule-based validation and Isolation Forest anomaly detection within an In-Extract, Transform, Load (In-ETL) pipeline. Incoming records are dynamically classified into Clean, Suspicious, and Rejected categories prior to warehouse storage. Evaluation utilizing a hybrid dataset incorporating 2,344 authentic commodity records from the Makassar City portal scaled to 100,000, demonstrated effective data triage, categorizing 88.11% as Clean. A baseline algorithmic comparison further justified the selection of Isolation Forest, which maintained near-instantaneous processing (0.14 seconds) compared to the severe latency bottlenecks of distance-based models like LOF and One-Class SVM. Compared to reactive Post-ETL methods, the In-ETL architecture reduced dirty data exposure by 79.11% and maintained 97.63% warehouse purity. These findings confirm that integrating preventative anomaly detection during the ETL transit phase significantly improves early-stage data quality and minimizes downstream errors. Future research should focus on expanding this architecture to accommodate complex unstructured data types and national-level portal integrations.