The Biggest Data Challenges for CEOs in 2022–2023

A survey of IT professionals showed that, in general, businesses are faced with five big challenges in data collection:

1. Inability to use gathered data (39% of the respondents).

2. Ineffective management of stored data (37%).

3. Insufficient security of the collected data (35%).

4. Uneven availability of disparate databases (30%).

5. Collecting only required data (36%).

These issues remain common to IT infrastructures around the world, although specific percentages vary by country and region. Two-thirds of respondents note that data security is insufficient: improving it is a key issue for improving management systems.

The amount of data is growing at a tremendous pace: according to analysts, in the next couple of years, the growth will be 42.2% per year. This is easily explained by the growing population of the planet, the rapid digitalization, the increasing popularity of teleworking of employees and remote management of business processes.

This is largely due to the fact that the data is not stored centrally, but distributed between cloud storage and edge environments. Accordingly, they become more difficult to manage.

With the introduction of neural networks, machine learning systems and the Internet of Things in various areas of human life and business, the development of 5G networks and user devices at the edge, and not in the data center, business decisions are often made in real time. Peripheral components of the network can be removed from the center by tens or even hundreds of kilometers.

There are many examples: unmanned delivery, mining platforms, smart home and smart office devices controlled from a smartphone, production sites in industrial plants, and much more.

Cloud storage accounts for 22% of the data volume, and only 9% is stored elsewhere. Experts predict that this distribution will not change significantly in the short term, and in the next year or two, corporate storage environments will be as fragmented as they are now.

The volume of data at peripheral objects is growing faster: about 36% of this information, after processing, moves from the periphery to the center, of which about 8% — in real time. Seagate analysts predict that in two years these numbers will be 57% and 16%, respectively. Accordingly, there will be a need for centralized management of distributed data. More specific tasks depend on the specific type of clouds.

Multi-cloud, private and public clouds: pros and cons

Many organizations store data in public cloud storage, available over the Internet (for free or for a fee) to everyone. Among the advantages of this type of clouds are:

• Rapid growth and scaling.
• Accessibility from any device in the IT infrastructure.
• Access to an extensive catalog of services.

However, with the growth and development of the company’s activities, such a solution is often not flexible enough. The most suitable option for business is using multi-cloud ecosystems and combined storage and data management solutions.

Multi-cloud systems combine public clouds from different providers with private ones available only to specific users. The advantages of private clouds are:

• Economies of scale under the control of the organization.
• Ownership of storage IP addresses to protect and control storage systems.
• The ability to frequently access large datasets.
• Greater confidentiality (which is important, for example, for medical or legal documents).
• The so-called hybrid clouds are also widely used, combining the resources of a private and public cloud within a single integrated infrastructure. This helps to establish interaction between individual data stores and manage them centrally through a single management portal. The main challenge here is connecting legacy systems to public clouds, which is easier to accomplish in a multi-cloud environment.

Experts say that the biggest challenge in the short to medium term will be managing enterprise data in multi-cloud ecosystems and hybrid clouds. And many management problems can be solved by implementing the DataOps methodology, linking data creators and consumers with related processes.

DataOps Methodology benefits

DataOps is a methodology that accelerates storage modernization and is based on interactions between creators and consumers. The last to speak are the employees of the company who are responsible for organizational issues: as a rule, these are the top management and the personnel who help them. They do not need the initial information, but the results of its analysis, which serve as the basis for making decisions.

Data creators can be both people (managers, analysts, IT professionals, and others) and digital devices. When analyzing information, the question often arises of which information should be used immediately and which should be sent for storage.

For example, if we are talking about a device in an IT infrastructure, then its technical characteristics may require instant analysis to coordinate work and forecasts for the near future, and information about its activity is already transferred to the storage.

Artificial intelligence (AI) and machine learning (ML) are widely used within DataOps: these technologies allow to establish relationships between data from the center, the periphery and cloud storage.

To obtain data arrays, a process is used, which is based on the ELT principle (Extract, Load, Transform — “extract, load, transform”). In the course of its work, disparate data from several sources are loaded into a single structure, which is structured and turns into clear information using AI: consumers can already work with it and make decisions based on it.

The main competitive advantage of DataOps is the ability to easily establish relationships between disparate data using a single tool. Without DataOps, such tasks require multiple tools at once, which complicates and slows down management.

According to Seagate, only 10% of companies surveyed have fully implemented DataOps. The largest percentage (12%) is observed in the mass media, the smallest (5%) in production. The effective use of the methodology is hindered by both technical problems and the human factor (competition between employees and teams: personnel unpreparedness for innovations). At the same time, the majority of respondents (89%) consider DataOps to be important for business development. Only 1% of the respondents do not attach importance to the methodology.

DataOps is most popular in China (the percentage of Chinese specialists who do not take the methodology into account turns out to be zero) and North America (USA and Canada). The need for it is especially high in the transport sector.

Since the survey was conducted at the beginning of 2020, before the COVID-19 pandemic, it can be concluded that currently the need to implement DataOps is even higher due to the transition of employees of companies around the world to a remote mode of work (which has already led to migration into cloud services). For Russia, this is relevant primarily for large cities characterized by a developed IT infrastructure and widespread use of “clouds”.

The competent implementation of DataOps contributes to the creation and development of artificial intelligence models and the widespread adoption of data analytics. Thanks to structured analytical information, companies gain competitive advantages in the market, increase profits and make interaction between employees in different departments more efficient, including those geographically distant from each other.