The more organizations that start to appreciate just how dependent they are on data to drive digital business transformation initiatives the more apparent it’s become how difficult it is to manage data.
A survey of 200 IT, data science, and data engineering professionals at organizations in North America with at least 1,000 employees conducted by IDG Research on behalf of Matillion, a provider of tools for managing cloud data warehouses, finds it takes about a week, on average, to aggregate and prep data so that it is useful for analysis. In fact, nearly half (45 percent) of time spent on data analytics projects is dedicated to data preparation.
Not surprisingly, the survey finds nearly all respondents (97 percent) are looking for ways to accelerate data transformation processes, with 60 percent citing scalability and flexibility as a challenge when preparing data for analytics
Overall, the top challenge cited by nearly half of respondents (47 percent) are issues pertaining to data control, followed by lack of a scalable, reliable technology platform to process large data sets (45 percent), lack of visibility and control of data silos when collaborating with business users (40 percent), having too many manual processes (38 percent), and challenges cleansing and preparing data (36 percent).
More than one-third (38 percent) are already using cloud data warehouses (CDWs), with 43 percent expecting to eventually have all of their data in the cloud. The rest plan to either rely on hybrid cloud models that leverage both cloud and on-premises data warehouses.
Keep an eye on data lakes
However, while usage of CDWs is widespread, only 16 percent currently use data lakes. More than half (56 percent) plan to use data lakes in the future, while another 26 percent are considering employing one. More than half of respondents (57 percent) said they leverage a hybrid cloud strategy for data management, while 22 percent are planning a multi-cloud strategy and 21 percent are planning on relying on a single cloud provider.
Finally respondents said data portability (57 percent), ease of onboarding (57 percent), and cost effectiveness (52 precent) are the top features required of an analytics platform. IT professionals specifically favored user-friendly (50 percent) and easier connections to data sources (50 percent) as top features, while data professionals cited time-to-value (57 percent) and self-service capabilities (51 percent).
Obviously, organizations are struggling with myriad data management issues, and there is no shortage of opportunities for managed service providers (MSPs). Digital business transformation initiatives are exposing issues ranging from processes that are either simply too slow or downright sloppy. Furthermore, data security issues that have often been left largely addressed, so chances are good there’s a high level of discomfiture inside organizations as more difficult questions are increasingly asked.
The need for speed when it comes to processing data has never been more critical. Digital business transformation initiatives depend on event-driven IT platforms that process data in near real time. If it takes a week or more just to prepare data, a digital business transformation initiative is doomed from the start. MSPs have a unique opportunity to present themselves as stewards of modern data management processes that will enable organizations to fulfill their real digital business transformation potential.
Of course, MSPs need to have the skills and expertise needed to manage data at the rate of speed now required. That may be a tall order for some MSPs but those that are not able to meet that challenge will soon find it increasingly difficult to remain relevant to their end customers.
Photo: Khing Choy / Shutterstock
