A Technical Architecture for the Integration of Knowledge Discovery in Databases and Data Warehousing Systems`
AUTHORS
Niamh O'Callaghan,Senior Research Fellow, Centre for Data Analytics and Knowledge Systems, Trinity College Dublin, Ireland
Eoin Gallagher,Senior Research Fellow, Centre for Data Analytics and Knowledge Systems, Trinity College Dublin, Ireland
ABSTRACT
The increasing demand for intelligent, data-driven decision-making has underscored the importance of integrating advanced analytical capabilities within enterprise data management systems. This paper presents a technical architecture that unifies Knowledge Discovery in Databases (KDD) with Data Warehousing (DW) systems to enable scalable, modular, and efficient extraction of actionable insights from large datasets. The proposed architecture addresses core integration challenges by organizing system components into five functional layers, encompassing data ingestion, transformation, storage, mining, and visualization. By embedding KDD processes directly into the data warehousing framework, the architecture supports continuous knowledge extraction while maintaining system performance and flexibility. The methodology includes system design, tool-based simulation, and performance evaluation using real-world and synthetic datasets to assess scalability, mining accuracy, and processing efficiency. Results demonstrate that the architecture achieves high mining accuracy, reduced data processing latency, and effective separation of concerns through modular components. Furthermore, the design accommodates near-real-time data flows, making it suitable for dynamic environments where timely insights are critical. The findings validate the feasibility and advantages of embedding intelligent analytical functions within warehouse infrastructures, bridging the gap between data storage and knowledge extraction. This work contributes a replicable model for organizations seeking to enhance their data architectures with integrated analytical capabilities. It lays the groundwork for future extensions involving cloud-native platforms and advanced AI-driven discovery techniques.
KEYWORDS
Knowledge Discovery in Databases (KDD), Data warehousing, Data integration architecture, Big data analytics, Enterprise decision support systems
REFERENCES
[1] X. Shu and Y. Ye, “Knowledge discovery: Methods from data mining and machine learning,” Social Science Research, vol.110, pp.102817, (2023). DOI:10.1016/j.ssresearch.2022.102817(CrossRef)(Google Scholar)
[2] A. Nanda, S. Gupta, and M. Vijrania, “A comprehensive survey of OLAP: Recent trends,” 2019 3rd International Conference on Electronics, Communication and Aerospace Technology (ICECA), pp.425–430, (2019). DOI:10.1109/ICECA.2019.8822203(CrossRef)(Google Scholar)
[3] T. Dubuc, F. Stahl, and E. B. Roesch, “Mapping the big data landscape: Technologies, platforms and paradigms for real-time analytics of data streams,” IEEE Access, vol.9, pp.15351–15374, (2021)
[4] A. Nambiar and D. Mundra, “An overview of data warehouse and Data Lake in modern enterprise data management,” Big Data and Cognitive Computing, vol.6, no.4, pp.132, (2022). DOI:10.3390/bdcc6040132(CrossRef)(Google Scholar)
[5] A. Oussous, F. Benjelloun, A. Ait Lahcen, and S. Belfkih, “Big data technologies: A survey,” Journal of King Saud University - Computer and Information Sciences, vol.30, no.4, pp.431–448, (2018). DOI:10.1016/j.jksuci.2017.06.001(CrossRef)(Google Scholar)
[6] Y. Si, J. Du, Z. Li, X. Jiang, T. Miller, F. Wang, W. J. Zheng, and K. Roberts, “Deep representation learning of patient data from electronic health records (EHR): A systematic review,” Journal of Biomedical Informatics, vol.115, pp.103671, (2020). DOI:10.1016/j.jbi.2020.103671(CrossRef)(Google Scholar)
[7] M. Chen, Q. Zhu, and Z. Chen, “An integrated interactive environment for knowledge discovery from heterogeneous data resources,” Information and Software Technology, vol.43, no.8, pp.487–496, (2001). DOI:10.1016/S0950-5849(01)00159-8(CrossRef)(Google Scholar)
[8] H. Hashim, “Hybrid warehouse model and solutions for climate data analysis,” Journal of Computer and Communications, vol.8, pp.75–98, (2020). DOI:10.4236/jcc.2020.810008(CrossRef)(Google Scholar)
[9] Y. Tang, G. T. Ho, Y. Lau, and S. Tsui, “Integrated smart warehouse and manufacturing management with demand forecasting in small-scale cyclical industries,” Machines, vol.10, no.6, pp.472, (2022). DOI:10.3390/machines1006047(CrossRef)(Google Scholar)
[10] M. H. Jarrahi, D. Askay, A. Eshraghi, and P. Smith, “Artificial intelligence and knowledge management: A partnership between human and AI,” Business Horizons, vol.66, no.1, pp.87–99, (2022). DOI:10.1016/j.bushor.2022.03.002(CrossRef)(Google Scholar)
[11] H. R. Nemati, D. M. Steiger, L. S. Iyer, and R. T. Herschel, “Knowledge warehouse: An architectural integration of knowledge management, decision support, artificial intelligence and data warehousing,” Decision Support Systems, vol.33, no.2, pp.143–161, (2002). DOI:10.1016/S0167-9236(01)00141-5(CrossRef)(Google Scholar)
[12] H. Foidl, V. Golendukhina, R. Ramler, and M. Felderer, “Data pipeline quality: Influencing factors, root causes of data-related issues, and processing problem areas for developers,” Journal of Systems and Software, vol.207, pp.111855, (2023). DOI:10.1016/j.jss.2023.111855(CrossRef)(Google Scholar)
[13] Z. Wang, Q. Shen, S. Bi, and C. Fu, “AI empowers data mining models for financial fraud detection and prevention systems,” Procedia Computer Science, vol.243, pp.891–899, (2023). DOI:10.1016/j.procs.2024.09.107(CrossRef)(Google Scholar)
[14] A. I. Saleh, A. H. Rabie, and K. M. Abo-Al-Ez, “A data mining based load forecasting strategy for smart electrical grids,” Advanced Engineering Informatics, vol.30, no.3, pp.422–448, (2016). DOI:10.1016/j.aei.2016.05.005(CrossRef)(Google Scholar)
[15] P. Sunhare, R. R. Chowdhary, and M. K. Chattopadhyay, “Internet of things and data mining: An application oriented survey,” Journal of King Saud University - Computer and Information Sciences, vol.34, no.6, pp.3569–3590, (2022). DOI:10.1016/j.jksuci.2020.07.002(CrossRef)(Google Scholar)
[16] V. Sharma, A. K. Tripathi, and H. Mittal, “Technological revolutions in smart farming: Current trends, challenges & future directions,” Computers and Electronics in Agriculture, vol.201, pp.107217, (2022). DOI:10.1016/j.compag.2022.107217(CrossRef)(Google Scholar)
[17] A. Saboor, M. F. Hassan, R. Akbar, S. N. Shah, F. Hassan, S. A. Magsi, and M. A. Siddiqui, “Containerized microservices orchestration and provisioning in cloud computing: A conceptual framework and future perspectives,” Applied Sciences, vol.12, no.12, pp.5793, (2021). DOI:10.3390/app12125793(CrossRef)(Google Scholar)
[18] L. Dinesh and K. G. Devi, “An efficient hybrid optimization of ETL process in data warehouse of cloud architecture,” Journal of Cloud Computing, vol.13, no.1, pp.1–12, (2024). DOI:10.1186/s13677-023-00571-y(CrossRef)(Google Scholar)
[19] M. E. Cambronero, M. A. Martínez, L. Llana, R. J. Rodríguez, and A. Russo, “Towards a GDPR-compliant cloud architecture with data privacy controlled through sticky policies,” PeerJ Computer Science, vol.10, pp.e1898, (2024). DOI:10.7717/peerj-cs.1898(CrossRef)(Google Scholar)
[20] T. L. L. Siqueira, R. R. Ciferri, V. C. Times, and C. D. de Aguiar Ciferri, “Benchmarking spatial data warehouses,” in Data Warehousing and Knowledge Discovery. DaWaK 2010. Lecture Notes in Computer Science, vol.6263, pp.50–61, (2010). DOI:10.1007/978-3-642-15105-7_4(CrossRef)(Google Scholar)
[21] J. Doncevic, K. Fertalj, M. Brcic, and A. Krajna, “Mask–mediator–wrapper: A revised mediator–wrapper architecture for heterogeneous data source integration,” Applied Sciences, vol.13, no.4, pp.2471, (2022). DOI:10.3390/app13042471(CrossRef)(Google Scholar)
[22] S. Trindade, L. F. Bittencourt, and N. L. D. Fonseca, “Resource management at the network edge for federated learning,” Digital Communications and Networks, vol.10, no.3, pp.765–782, (2024). DOI:10.1016/j.dcan.2022.10.015(CrossRef)(Google Scholar)