{"id":3122,"date":"2024-04-01T23:14:49","date_gmt":"2024-04-01T21:14:49","guid":{"rendered":"https:\/\/thedatastory.nl\/?p=3122"},"modified":"2024-04-01T23:14:50","modified_gmt":"2024-04-01T21:14:50","slug":"data-quality-is-not-difficult-how-automation-simplifies","status":"publish","type":"post","link":"https:\/\/thedatastory.nl\/en\/data-stories\/data-quality-is-not-difficult-how-automation-simplifies\/","title":{"rendered":"Data Quality is not difficult – how automation simplifies"},"content":{"rendered":"

When companies start collecting more data, data quality eventually becomes a topic of discussion. With more plants in your garden, maintenance is a larger responsibility. It is good to be cautious, but the problem quickly grows out of proportion: Once companies decide to start measuring data quality, they might appoint a whole team to take care of it. When they discover that they do not have a company-wide data quality standard, they will try to come up with one. Eventually, an entire data governance team might rise from the ground, which will explore a wide range of tools. <\/p>\n\n\n\n

At some point, people might start wondering \u201cwhat actually is data quality?\u201d, followed by countless meetings to define it. You get the gist – companies are overcomplicating data quality which costs money, time and other resources. However, data quality is actually very straight-forward. Sometimes there is no need for a full team of gardeners to maintain a high level of data quality. <\/p>\n\n\n\n

This blog will demonstrate why data quality doesn\u2019t have to be so difficult: it will show how automation can help guarantee the condition of the majority of your data. First, the reasons to do so will be discussed. Next, the key ingredients for data quality will be discussed. Using these, we will go over an algorithm to automate data quality.<\/p>\n\n\n\n

\"Data<\/figure>\n\n\n\n

Why data quality?<\/strong><\/h2>\n\n\n\n

There are numerous reasons to strive for a high level of data quality. For example, high-quality data is essential for Artificial Intelligence (AI) and Machine Learning (ML) models. Check out our latest blog<\/a> to learn more about the reasons for this. According to Gartner<\/a>, 85% of AI or ML projects fail to move past preliminary stages due to poor data quality. <\/p>\n\n\n\n

Not only do projects not get to see the light of day, if they do low-quality data can come with a cost: IBM estimated the annual cost of \u201cbad data\u201d to be $3.1 trillion, in the US alone. Similarly, in the MIT Sloan Management Review of 2017, the authors stated that \u201cthe cost of bad data is an astonishing 15% to 25% of revenue for most companies\u201d. It seems like common sense to value data quality – it saves time, reduces cost and improves project outcome. As stated in our previous blog, Garbage In Garbage Out definitely applies to data in business processes. The wrong crops in your garden will naturally yield the wrong produce. Now that we understand why we should maintain our garden, let\u2019s see what requirements there are.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

What defines data quality?<\/strong><\/h2>\n\n\n\n

The quality can be measured through 6 dimensions:<\/p>\n\n\n\n