Enhanced data quality: Building foundations with our data profiling capability and operating model
Effective data management is critical to achieving our ambitions of becoming a data driven organisation and a key part of data management is data quality.
In line with the Government Data Quality Framework, we aspire to have robust, consistent and objective approaches to document, understand and improve data quality both proactively and retroactively.
To address the proactive aspect, we established a data profiling capability and operating model within the business which utilises a combination of technology, enhanced processes, and skilled colleagues to deliver successfully.
As a new capability, we typically emphasise putting a scale to known business issues, but we have also taken a risk-based approach where we seek to identify unknown data quality issues that may be lurking.
Since establishing the team and embedding the technology we have created over 700 customised rules, conducted data profiling on 14 of our critical data assets and identified more than 150 improvement actions.
Several key factors have contributed to the success of this initiative:
- focus on striking a balance between people, processes and technology to ensure an effective end-to-end data profiling process
- a strong emphasis on collaborative working with stakeholders from across the business and beyond including the creation of safe, supportive relationships which enable colleagues to openly share known issues and data quality concerns
- the adoption of a low code tool enables us to recruit colleagues who are passionate about data and have the softer skills needed to be great collaborators then quickly up skill them to utilise the technical solution
- embracing a continuous improvement culture in which all team members are empowered to identify areas for improvement and suggest solutions which help enhance the effectiveness and efficiency of our work
There is an inescapable irony in measuring success by finding, scaling, and addressing failure, so our greater ambition is to ensure that good data governance is factored into new services and system replacements to help reduce the risk of future data quality issues occurring. We are also working on the creation of a data model and dashboard that will allow us to showcase the holistic impact of our data profiling projects so far.