Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
March 15, 2018
prev slideNext slide

IT Decision-Makers Must Focus on the Importance of Big Data Staffing

While there’s no overstating the importance of big data, this technology comes with a significant consideration for acquiring new skills in your IT organization. Also, in an age of automation, there is anxiety that advancements in technology and the continued evolution of big data and artificial intelligence will lead to the replacement of people by machines.

Certain types of jobs, of course, will be in jeopardy. Oxford University researchers estimated that 47 percent of jobs in the U.S. are at risk of being automated during the next 20 years, particularly in transport, logistics, and automation. Analytics specialist Bernard Marr even stated that the role of data scientist is at risk because new machine-learning algorithms can identify patterns, interpret information, and produce visualizations.

It’s important to note that such technology-inspired shifts are nothing new. Across all types of revolutionary change, from industrial to informational, some occupations have always been left by the wayside. Think, for example, of the move in transportation from horses to cars and the change in associated occupations. Then think of the widespread use of social media during the past decade or so and the development of new roles, such as digital marketing, across platforms such as LinkedIn and Facebook. The same premise will be true in the next phase of change driven through big data and artificial intelligence: automation will eliminate some jobs, and it will create new ones.

The Importance of Big Data Drives Change in the Digital Age

While the power of automation is significant, machines do not operate in isolation. IT decision-makers must recognize the importance of big data, but they must also find ways to help their staffs to excel in the digital age. Businesses that focus on different types of capability, management, training, and strategy will develop employees who create business value from data.

Different Types of Capability

Data analysts have traditionally focused on numbers. That quantitative analysis covered key business measures, such as operations, finances, and efficiencies. In the digital age, according to Gartner, organizations are looking for a new blend of skills to deal with the importance of big data. This blend, rather than being solely concentrated on hard facts and figures, includes softer, social characteristics. Successful businesses now need to understand the qualitative data behind experiences and decision-making. A retailer, for example, might focus on the minimum number of clicks required to make a purchase, and the maximum before a customer loses interest.

This shift does create some uncertainty for data scientists. Rather than a highly rigid skill set, organizations need flexible employees who can hone their capabilities as the required business hypotheses change. Modern data scientists must be agile and comfortable with a DevOps style of working.

Different Types of Management

There has been a shift in the management of technology, from a technical dark art to the provision of digital business services. The decision-making process associated with these services—and the skills associated with implementation—have become distributed across the organization.

Rather than being the sole preserve of the IT department, technology—and the data it produces and consumes—is owned by many people in a business, according to Forrester. Just as this consumption of data analytics takes place across lines of business, the management of information—particularly regarding governance and security—must also take place at multiple points.

In this new, decentralized age of data management, a linear approach will not suffice. Governance and security become everyone’s responsibility. Organizations must create an iterative way of working, with a close relationship between IT and the rest of the business.

Different Types of Training

Modern businesses need a new type of data scientist, and the importance of big data means talented workers are not only in high demand but also tough to source. Organizations are prepared to pay a high premium for in-demand skills—such as Hadoop, Spark, SQL, and visualization—that will help exploit informational resources.

However, while sourcing is one way to fill the skills gap, The Economist noted that you shouldn’t assume your existing workers can’t be retrained and revitalized. Remember that your retained talent understands your business and its priorities. The key to success is to pick people who can adapt, and then train these individuals for data-led change.

Implement a program to train people who can develop new skills quickly. Rather than creating tomorrow’s legacy, use techniques such as e-learning, self-learning, and communities of knowledge to develop an iterative approach to education. In short, create a team that embraces constant transformation.

Different Types of Technology

Staffing isn’t the only area that requires careful consideration. IT decision-makers must also pay attention to the advanced technology that is brought into the business on an ongoing basis. From the Internet of Things to automation, fresh technology creates new data management issues.

To be successful, IT leaders must develop a global data management strategy to help the business cope with digital change. This strategy should encompass a range of critical areas, such as governance, security, scalability, and support, including service levels, backups, and disaster recovery scenarios. Put the goal posts in place to help them hit their targets as they work with data at scale, creating new value for the business.

New Opportunities for Data Professionals

A new cadre of jobs is being created. Rather than sounding a death knell for your existing workforce, smart IT decision-makers will embrace the change and look at ways to create and reskill their data professionals. In short, a new era of skills in the digital era is creating a fresh range of opportunities for talented data professionals.

Learn more about how automation can unlock the potential for predictive analysis and other advanced capabilities for data scientists.


Will Dailey says:

Across the globe there is an accelerating need for a complex set of skills required to design, deploy, operate and optimize a our Big Data solutions. Open source technology offers many advantageous, but one of them is not simplicity and adherence to uniform standards. It takes a Big Data team, perhaps not a large team, but a team of Hadoop operators and administrators, data engineers, data scientists, and adapt business users to gain maximum value from our solutions. Big Data Teams, in particular those organizations just embracing Big Data, require immersive education to ensure success. My comment to clients is consistently “You do not have a personnel gap, you do not even have a skills gap, you only have an education gap.” And this gap can be filled with the right educational plans for your IT staff.

Leave a Reply

Your email address will not be published. Required fields are marked *