esc

Type at least two characters to start searching.

↑↓ select↵ openesc close

Share

Export Citation

APA
MLA
Chicago
Harvard
Vancouver
BIBTEX
RIS
Universitas Hasanuddin
Research output:Contribution to journal›Article›peer-review

In-ETL Architecture with Isolation Forest-Based Data Triage for Improving Open Government Data Quality

Asdar M.

2026 IEEE International Conference on Artificial Intelligence and Mechatronics Systems Aims 2026

Published: 2026

Abstract

Implementing Open Government Data (OGD) in Indonesia faces data quality challenges due to inconsistencies and anomalies across Regional Apparatus Organizations. This study proposes a proactive approach by integrating multi-stage rule-based validation and Isolation Forest anomaly detection within an In-Extract, Transform, Load (In-ETL) pipeline. Incoming records are dynamically classified into Clean, Suspicious, and Rejected categories prior to warehouse storage. Evaluation utilizing a hybrid dataset incorporating 2,344 authentic commodity records from the Makassar City portal scaled to 100,000, demonstrated effective data triage, categorizing 88.11% as Clean. A baseline algorithmic comparison further justified the selection of Isolation Forest, which maintained near-instantaneous processing (0.14 seconds) compared to the severe latency bottlenecks of distance-based models like LOF and One-Class SVM. Compared to reactive Post-ETL methods, the In-ETL architecture reduced dirty data exposure by 79.11% and maintained 97.63% warehouse purity. These findings confirm that integrating preventative anomaly detection during the ETL transit phase significantly improves early-stage data quality and minimizes downstream errors. Future research should focus on expanding this architecture to accommodate complex unstructured data types and national-level portal integrations.

Other files and links

Fingerprint

Data qualitySciences
Computer scienceSciences
ArchitectureSciences
Government (linguistics)Sciences
Quality (philosophy)Sciences
Isolation (microbiology)Sciences
Open dataSciences
TriageSciences
Computer securitySciences
Open governmentSciences
BusinessSciences
Key (lock)Sciences
Systems architectureSciences
ServerSciences
Process managementSciences