Think Big… Right Start Big Data Projects
Think Big Analytics, a Hortonworks systems integration partner has been helping customers navigate the complex world of Hadoop successfully for the past three years. Over the years they have seen it all and have developed one of the most mature Hadoop implementation methodologies known. Recently, we asked Ron Bodkin, Founder and CEO of Think Big Analytics to share some insight.
What are the “Must-Dos” Before Starting a Big Data Project?
To start, and to take into consideration throughout each stage are these three “Must –Dos”:
1. Test and learn
An agile approach with rapid releases enables organizations to fine-tune their projects while they’re in progress. Traditional legacy systems were better suited to a “waterfall” approach, where technology was introduced all at once. Big Data projects should focus on specific business goals and allow cross-pollination of ideas to better understand what’s possible, making a “rapid release” approach much better.
2. Incremental adoption
Build a center of competency and cross-pollinate expertise among business experts, data scientists, and data engineers. This approach enables business units to leverage a common talent pool and a shared approach, eliminating the risk of data silos, providing for common governance, and avoiding redundant storage and processing by different departments.
3. Change management
Think about key stakeholders for the initiative, understand their concerns, get their buy in and invest in early pilot systems that demonstrate the value that can be generated through a Big Data investment.
How does a customer think through Adoption of Hadoop and Big Data?
To help customers through this process for successful implementation, Think Big developed a maturity model that outlines four stages an organization should step through. Most important is how businesses should respond to outcomes to move forward at each stage.
1. Contain Costs and Scale
This stage of adoption of Big Data is about laying a foundation where data can be stored and accessed by a larger audience, allowing for better scale and favorable economics to support creating value at a greater volume, variety and velocity than previously possible. To succeed in this stage:
- Avoid the “Big Data plateau.” If you store and process data sets that are only useful only for generating reports or feeding a data warehouse, it can be hard to engage the business. Look to include a combination of data sets that will allow future business value, not just IT efficiency, e.g., combining web, social, product log, text, geo, mobile or other new data sets with structured relational data about customers and transactions. Prepare for agile analytics by getting the business engaged in the conversation in planning discussions early on.
- Don’t wait. Instead of trying to sell a big project to get started, leverage existing budgets and carve out some low hanging fruit that will provide immediate value.
- Get help in planning and executing. Big data requires different skills, different data processing, different ways of storing and modeling data, and different governance skills. It’s important to do more than just train teams – it takes time to learn the patterns and practices so getting experienced help is crucial for early success.
- Hold your ground. New technologies can be frustrating to bring online and are often abandoned or stall out. Investment of time and budget in a Big Data project is necessary and it is important to stay the course.
2. Agile Analytics and Insights
The next stage of adoption of Big Data is about empowering the business to get meaningful insights in a much more agile way, allowing answers to data in hours not weeks. To build on the success of an initial Big Data platform it’s important to:
- Pick the right pilot: low hanging fruit with meaningful impact, not a “castle in the sky”. Focus on projects that can generate business value in eight week so key constituents are able to see for themselves the value of Big Data.
- Pick up steam after success. Following the launch of a first successful project, follow a portfolio approach: invest in smaller projects that build on the existing applications, which can result in more value as well as larger, greater projects. Bring new tools and data sets to extend systems incrementally – instead of a “big bang” delivery, think about the smallest set of technologies and data sets that can produce results, then keep extending incrementally based on business needs.
- Stop speculating. Big Data lets you stop wasting time and energy in speculative projects that are meant to parse, summarize, and organize data in data warehouses and data marts. Instead, you can explore and investigate new concepts with raw data, letting data scientists and analysts test out and refine approaches quickly, then build standardized support AFTER an analysis or dimension is proven in value.
- Remember: Data is not the enemy. Traditionally companies have a bunker mentality to the increasing deluge of data, finding ways to condense and delete data. With Big Data, the better mindset is to embrace data as an asset, and retain it for analysis. Furthermore, there’s tremendous value in seeking out new data sets from external sources (vendors, partners, government, open data sets and otherwise) to provide better insights.
3. Business Innovation
This stage of adoption is about driving business innovation by integrating Big Data predictive analytics into business processes, to automate decision processes and even to begin gaining insights that enhance products. To make this work it’s important to:
- Emphasize your goals so data scientists and the business have a shared understanding of the problem. This lets you invest effort in activities that will move the needle, rather than working on second order problems.
- Test and test again. Test quickly – there’s no substitute for real-world experience when testing out new ideas. With Big Data it’s possible to prepare many tests and use A/B testing principles to run parallel experiments at small scale, so you can double down on approaches that are working quickly.
- Invest in data. Companies often get stuck with limited visibility so in order to avoid this, invest in data. In other words,, instrument your applications effectively and run control groups so you can measure improvement and get insight into how alternative strategies fare. Having a lab sandbox to simulate expected results, based on the best data available is important to keep innovating.
4. Organizational Transformation
The final stage of adoption is about transforming your culture so data is central to decisions and empowers new opportunities. To allow this:
- Don’t accept assertions, require analysis and evidence. To instill a data-driven culture: it’s important reward those who make valuable discoveries based on your data, and challenge intuition by asking: “How can we measure that”?
- Think outside the box. Big Data is an exciting contributor to reinventing the way things “used to be done”. Integrate new data, simulate and measure throughout the discovery, assessment and refinement of new products and features. By doing so, organizations are opening themselves up to a green field of opportunity.
In conclusion, Big Data represents a major opportunity for the enterprise. The journey will take time but by making the right choices you can get value quickly and avoid pitfalls. By moving forward thoughtfully and with determination, you have the opportunity to gain significant advantages from this major new business innovation.
A great way to get started with a Big Data project is Think Big’s Jumpstart package.
Thank you to Think Big Analytics for your contribution. Visit Think Big Analytics for further information.
Get Started using Hadoop to Analyze Data. This guide includes tutorials, videos and advice on integrating Hadoop with popular analytics packages.