Autonomous Data Engineering

Authors

Dinesh Thangaraju

Abstract

The increasing complexity of modern data systems has positioned Artificial Intelligence (AI) as a transformative force in data engineering. AI-powered tools and frameworks are streamlining data pipeline orchestration, schema creation, and quality assurance, enabling enterprises to enhance the productivity of data engineers, increase operational speed and agility, and reduce costs. However, the adoption of AI in data engineering also introduces risks related to data security, bias, and compliance that require careful management.

This paper explores how AI is reshaping data engineering, focusing on autonomous data engineering, real-time anomaly detection, and self-service analytics. It highlights the benefits of integrating AI into data workflows while addressing the associated risks. A technical framework is proposed to implement AI-driven data engineering, supported by metrics to evaluate its effectiveness.

Keywords

Data Engineering, Artificial Intelligence, Autonomous Data Engineering, Data Pipelines, Data Governance, Anomaly Detection, AI-Driven Automation

Autonomous Data Engineering. Dinesh Thangaraju. 2024. IJIRCT, Volume 10, Issue 2. Pages 1-12. https://www.ijirct.org/viewPaper.php?paperId=2501109