The CIO’s playbook for AI data management success 

Darren Cunningham at Komprise explains why AI demands a new partnership between IT infrastructure and data leaders

 

As AI takes centre stage in the workforce and increasingly drives strategic initiatives, CIOs yet again find themselves tossed in the currents of disruption. Their roles are quickly evolving along with those of their staff. “CIOs must move beyond technology facilitation and take the lead in AI governance, cyber-security, skills development, sustainability, regulatory compliance, and technical debt management,” according to IDC analyst Daniel-Zoe Jimenez, in a recent IDC blog.

 

A central goal to AI success across any organisation is developing the right data infrastructure; this includes the people who can consistently achieve a high degree of data quality, data availability, data safety, performance and economic viability. 

 

Now that unstructured data makes up an estimated 90% of enterprise data and is the vital source for AI, the game plan is changing for key players in the equation of AI execution: data storage teams and analytics platform teams are now tasked with preparing and delivering the right data to AI models. 

 

 

Data use versus data value

Historically, storage teams have focused on data use by managing data availability, performance, cost, capacity and data protection with success measured by IOPS, uptime SLAs, and cost metrics. Data platform teams are concerned with discovering data value and rely on access to usable, clean, and relevant data. Until recently, they’ve worked primarily with structured and semi-structured data sources from applications and databases for their data warehouses and data lakes. 

 

Data teams today are discovering that leveraging unstructured data for AI is difficult and much different from BI projects using ETL because the process is no longer linear. They can’t just move it to a data lake and process it.  It’s too large and expensive to do so, and furthermore, AI runs in many different locations and applications. Moving large data swamps around is not practical nor even possible. 

 

What data teams need is help from their data storage counterparts to curate data at the source, enabling them to copy the right data for each project to the appropriate AI tool. That requires knowledge of unstructured data content to accurately filter it and find desired data sets.

 

 

From ETL to metadata workflows with governance

The new imperative is to filter, curate, and deliver only the right data into a data workflow before any processing or AI model training begins. It requires automated metadata enrichment and orchestration. It also requires the ability to filter out sensitive data from the workflow. 

 

Storage and data teams, under the leadership of the CIO, will need new tools and processes (including human verification) to efficiently discover, classify and move the right data sets to AI for each project, while preventing any potential quality issues and security violations. Historically siloed, these groups will now need to work together more closely to ensure safe, high-quality AI data pipelines. 

 

Storage teams will need to evolve from infrastructure maintenance into data service providers to help data analytics teams as they gather the requirements and design processes for AI. Data platform teams will need to build common metadata definitions with storage teams and collaborate on an inclusive data governance strategy incorporating unstructured data.

 

To collaborate effectively, data teams will need to understand: 

• Basic file and object storage protocols (SMB, NFS, S3)
• What unstructured data exists and is usable by AI
• Storage access patterns and data tier and archiving strategies
• Metadata enrichment strategies and tactics
• Governance policies around sensitive and regulated data 

Data storage teams will need to understand:   

• Data modelling, pipelines, and AI data preparation requirements
• How to create metadata that is useful for data teams
• New use cases that drive demand for unstructured data, such as AI customer chatbots or department-specific GenAI tools
• Evolving performance, security and access needs for AI workloads  

 

How AI impacts the CIO role

To support emerging AI business models and changing IT roles, CIOs will need to supply the right tools for these new requirements, including:  

• Unified visibility across all unstructured data: Data scientists need to discover relevant data regardless of where it lives across departmental and storage silos. Shared reports on data usage, growth, file types, top owners and top metadata categories help relay the critical metrics.
• Automated metadata enrichment and tagging: Storage and data teams alike benefit from automated, AI-powered metadata capture, including content-aware tags, PII detection, access patterns and usage history. Storage stakeholders can use this data to ensure that they are adequately protecting sensitive and regulated data and right placing data for cost optimisation, while data teams appreciate the content-aware tags for rapid data curation.
• Intelligent data ingestion pipelines: Instead of duplicating petabyte-scale data sets for AI use, institute policy-based movement of the right data sets continuously based on metadata tags.
• Governed self-service access: Empower data teams with self-service search, and access to unstructured data sources.
• AI data governance checks and balances:  Through tools and manual inspection, teams must be empowered to segregate sensitive and proprietary data into a private, secure domain which restricts sharing with commercial applications, track the lineage of data, check data for biases and errors and facilitate audit trails of any data used in an AI tool and demonstrate that your organisation is complying. 

 

Data: a shared enterprise asset

For AI to deliver on its promise, structured and unstructured data must be treated as a shared enterprise asset, not the domain of isolated teams. If storage and data teams remain misaligned, organisations risk: 

• Wasting valuable data locked in silos
• Creating inefficient, costly and duplicative data pipelines
• Violating governance and compliance policies
• Slowing AI model development and time to value 

The future of global data strategy in the enterprise relies upon how well teams across IT and departmental boundaries can work together to meet the CXO-level goal of using AI safely and ingeniously to deliver profoundly better products, services and business outcomes. 

 

---

 

Darren Cunningham is the VP of Marketing at Komprise. Read more about their new preprocessing model for filtering unstructured data here. 

 

Main image courtesy of iStockPhoto.com and NicoElNino