Original Study

Open Access

|

Peer-reviewed

Data Observability: Ensuring Trust in Data Pipelines

Krishna Prasanth Brahmaji Kanagarla

DIP: 18.02.031/20240901

DOI: 10.25215/2455/0901031

Received: March 05, 2024; Revision Received: March 18, 2024; Accepted: March 30, 2024

Abstract

Data observability is the key process you need to implement to guarantee the consistency of the data pipeline in terms of credibility. The features of data quality, data freshness, lineage, and schema changes checked in real-time help prevent problems before they accumulate, and impact the data pipeline. As the field of data observability is still in its infancy, this paper aims at identifying which components it could be composed of, what tools it could ideally comprise, and what value it can bring to businesses, with an accentuation of recommendations for its implementation in contemporary data workloads.

Keywords

The author(s) appreciates all those who participated in the study and helped to facilitate the research process.

The author(s) declared no conflict of interest.

This is an Open Access Research distributed under the terms of the Creative Commons Attribution License (www.creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any Medium, provided the original work is properly cited.

© 2024, Kanagarla, K.P.B.

Responding Author Information

Krishna Prasanth Brahmaji Kanagarla @ msrprojectshyd@gmail.com

Related Content
Data Observability: Ensuring Trust in Data Pipelines

Total Download: 1 | Total View: 34

PlumX Matrix

Plum Analytics uses research metrics to help answer the questions and tell the stories about research. Research metrics that immediately measure awareness and interest give us new ways to uncover and tell the stories of research.

Dimensions Matrix

Dimensions is a next-generation linked research information system that makes it easier to find and access the most relevant information, analyze the academic and broader outcomes of research, and gather insights to inform future strategy. (digital science)

Article Overview

ISSN 2348-5396

ISSN 2349-3429

DIP: 18.02.031/20240901

DOI: 10.25215/2455/0901031

Published in

Volume 09, Issue 1, January – March, 2024

  • 1Downloads
  • 0Citations
  • 34Views
  • 0Likes
Download PDF
How to Cite
Print
Share
RSS
Follow by Email
Facebook
Twitter
  1. Shankar, S. and Parameswaran, A., 2021. Towards observability for production machine learning pipelines. arXiv preprint arXiv:2108.13557.
  2. Rupprecht, L., Davis, J.C., Arnold, C., Gur, Y. and Bhagwat, D., 2020. Improving reproducibility of data science pipelines through transparent provenance capture. Proceedings of the VLDB Endowment, 13(12), pp.3354-3368.
  3. Li, B., Peng, X., Xiang, Q., Wang, H., Xie, T., Sun, J., & Liu, X. (2022). Enjoy your observability: an industrial survey of microservice tracing and analysis. Empirical Software Engineering, 27, 1-28.
  4. Therrien, J. D., Nicolaï, N., & Vanrolleghem, P. A. (2020). A critical review of the data pipeline: how wastewater system operation flows from data to intelligence. Water Science and Technology, 82(12), 2613-2634.
  5. Fedushko, S., Ustyianovych, T., & Gregus, M. (2020). Real-time high-load infrastructure transaction status output prediction using operational intelligence and big data technologies. Electronics, 9(4), 668.
  6. Karumuri, S., Solleza, F., Zdonik, S., & Tatbul, N. (2021). Towards observability data management at scale. ACM Sigmod Record, 49(4), 18-23.
  7. Seedat, N., Imrie, F., & van der Schaar, M. (2022). Dc-check: A data-centric ai checklist to guide the development of reliable machine learning systems. arXiv preprint arXiv:2211.05764.
  8. Hurl, B., Cohen, R., Czarnecki, K., & Waslander, S. (2020, October). Trupercept: Trust modelling for autonomous vehicle cooperative perception from synthetic data. In 2020 IEEE Intelligent Vehicles Symposium (IV) (pp. 341-347). IEEE.
  9. Burgess, Mark, and Andras Gerlits. “Continuous integration of data histories into consistent namespaces.” arXiv preprint arXiv:2204.00470 (2022).
  10. Schnabel, G., Sjöstrand, H., Hansson, J., Rochman, D., Koning, A. and Capote, R., 2021. Conception and software implementation of a nuclear data evaluation pipeline. Nuclear Data Sheets, 173, pp.239-284.
  11. Sundaram, A., Abdel-Khalik, H.S., Roberson, D. and El Hariri, M., 2022. Data recovery via covert cognizance for unattended operational resilience. Progress in Nuclear Energy, 151, p.104317.
  12. Hartati, S., 2022. Innovative Approaches to Monitoring in Enterprise Production Systems. Sage Science Review of Applied Machine Learning, 5(1), pp.81-96.
  13. Shankar, S., Garcia, R., Hellerstein, J.M. and Parameswaran, A.G., 2022. Operationalizing machine learning: An interview study. arXiv preprint arXiv:‌2209‌.09‌‌125.
  14. Bento, A., Correia, J., Filipe, R., Araujo, F. and Cardoso, J., 2021. Automated analysis of distributed tracing: Challenges and research directions. Journal of Grid Computing, 19(1), p.9.
  15. Nestorov, A.M., Berral, J.L., Misale, C., Wang, C., Carrera, D. and Youssef, A., 2022, November. Floki: a proactive data forwarding system for direct inter-function communication for serverless workflows. In Proceedings of the Eighth International Workshop on Container Technologies and Container Clouds (pp. 13-18).
  16. van der Goes, M., 2021, September. Scaling enterprise recommender systems for decentralization. In Proceedings of the 15th ACM Conference on Recommender Systems (pp. 592-594).
  17. Munoz-Arcentales, A., López-Pernas, S., Pozo, A., Alonso, Á., Salvachúa, J. and Huecas, G., 2020. Data usage and access control in industrial data spaces: Implementation using FIWARE. Sustainability, 12(9), p.3885.
  18. Mora-Cantallops, M., Sánchez-Alonso, S., García-Barriocanal, E. and Sicilia, M.A., 2021. Traceability for trustworthy AI: a review of models and tools. Big Data and Cognitive Computing, 5(2), p.20.
  19. Patel, P. and Uddin, M.N., 2022. AI for algorithmic auditing: mitigating bias and improving fairness in big data systems. International Journal of Social Analytics, 7(12), pp.39-48.
  20. Perrault, A., Fang, F., Sinha, A. and Tambe, M., 2020. Artificial intelligence for social impact: Learning and planning in the data-to-deployment pipeline. AI Magazine, 41(4), pp.3-16.
  21. Haider, L. 2021. Artificial intelligence in ERP (Bachelor’s thesis). Metropolia University of Applied Sciences, Finland.
Make a Submission