Coupled with rising interest in artificial intelligence (AI), the need to derive business value from data is driving organizations to create data operations (DataOps) functions that should create additional opportunities for managed service providers.
A survey of 266 data professionals from more than 25 industries conducted by Nexla, a provider of a platform for managing data flows, finds 73 percent of respondents say their organizations will be investing in DataOps in 2018. DataOps is an attempt to build scalable, repeatable, and predictable data flows for data engineers, data scientists, application developers, and business users. DataOps tasks include integrating with data sources, performing transformations, converting data formats, writing or delivering data to its required destination, monitoring, governance, and security.
Similar in concept to DevOps, building a DataOps team is as much about culture within an organization as it is about products and technologies. In that regard, DataOps borrows many of the IT principles developed for DevOps and applies them to data management.
The rise of DataOps should create an opportunity for MSPs to build a new practice. The core issue that many organizations face is that they are spending more time managing data than analyzing it. The Nexla survey finds data professionals are only spending 14 percent of their time on analysis. The rest of their time is spent on required but low value-add tasks like data integration, data cleanup, and troubleshooting.
“The core issue that many organizations face is that they are spending more time managing data than analyzing it.” #DataOps @SmarterMSP
Data engineers report spending 18 percent of their time on troubleshooting, which equates to 9.3 weeks a year. The top two least enjoyable tasks cited by survey respondents were data cleanup (34 percent) and troubleshooting (24 percent).
A full 50 percent of the respondents said there are simply not enough backend resources and data engineers to support all the scientists, engineers, and end users trying to access data. Issues that need to be addressed include data format consistency (39 percent), integration (36 percent) and data pipeline reliability (35 percent), external data access (32 percent), and internal data access (19 percent).
A full 29 percent reported that the amount of data their organization is trying to manage is growing at a rate of more then 100GB a day. A full 85 percent of respondents said investments in machine learning algorithms and related AI technologies is a driving force behind the need to collect, storage, and manage massive amounts of data.
How MSPs can add value
The survey makes is clear respondents are hoping that advances in automation will solve their data problems. But, automating flawed processes doesn’t usually result in the desired outcome. Most organizations will require some external expertise to, at the very least, help them get their data in order, let alone manage it consistently.
That’s where MSPs have a significant opportunity to add value within the context of a larger DataOps initiative. Most MSPs already have a lot of experience managing data at scale. Making the leap to building a full DataOps practice based on that expertise is not that far a jump. In fact, if data is truly the new oil, then the organizations that refine that data are likely to be among the most valuable companies on the planet.