Hadoop big data gets personal
One of the reasons Hadoop big data analysis is particularly valuable is that much of it is fundamentally about people – what they do, what they buy, what they think about or what they want, etc. Analysis gleaned from big data offers a clearer picture of human conditions and enables organizations to anticipate needs and respond to wants. Many big data projects offer a more objective view of the way humanity functions and can trigger insight into how to make improvements.
The newest trend in big data insight is analysis directly targeted at people, reported Bloomberg Businessweek. In particular, 'People Analytics' is an approach through which organizations turn the focus inward and use big data comprised of external research and intercompany observations to make operational improvements and interact better with employees.
"Because most communication and collaboration happens face to face, the data are critical for people analytics to take that next leap forward and become a transformative organizational tool," wrote Businessweek contributor Ben Waber.…
Bringing big data to the racing world
NASCAR may dominate the U.S. racing scene, but elsewhere in the world Formula One is king. From Italy to China, F1 is viewed as the pinnacle of competitive motor sports, drawing hundreds of millions of viewers and generating massive revenue streams. According to the latest figures provided by the organization's global broadcast report, worldwide viewership reached 515 million in 2011, Reuters reported. After factoring in race hosting and television airing rights fees, trackside advertising and sponsorship revenue and ticket sales, the annual revenue generated by F1 totals approximately $1.5 billion, according to Autoweek.
With that much at stake, it's no wonder that the companies responsible for building race cars would take every available advantage to improve their vehicles' performance. In the case of Britain's McLaren Group, that includes deploying big data analytics to get the most out of their automobiles, ZDNet reported. The company's researchers outfitted each of its race cars with approximately 160 sensors that typically create one gigabyte of raw data during each event.…
Improving movie scripts with data analytics
Over the past few decades, Hollywood producers have increasingly become dependent on blockbuster movies to generate the revenue streams they require to rationalize investing millions of dollars into a single film, with blockbuster movies like "The Avengers" and "The Dark Knight Rises" forming the backbone of the filmmaking industry for some time now. According to the Economist, successful releases aimed at a wide audience were in large part responsible for box-office revenues reaching a record $10.8 billion in 2012. Individual movies are raking in more money than ever before. The recently released "Iron Man 3" has already earned $175.3 million after the first three days of its North American release, CNN reported. Last year's top earner, "The Avengers," made $623.4 million in the United States alone.
The rising costs of film development
However, the cost of funding these projects has skyrocketed as well. For example, "Iron Man 3" cost Disney approximately $200 million, according to CNN.…
Reduce travel stress with big data
Recent developments in big data applications use information about consumer habits and demographic trends to develop stress-reducing solutions for travelers. Travel management company CWT recently developed the Travel Stress Index, an algorithm-based tool that they hope will translate information into actionable recommendations, reported InformationWeek’s Ellis Booker.
“We had the transactional data and we had some traveler profile data, but it was scattered and we had to bring them together,” Catalin Ciobanu, CWT big data analyst, told InformationWeek.
Harnessing big data for pointed results involved a multi-faceted approach. They gave surveys to over 7,000 travelers from a variety of backgrounds, who were asked to rank 33 activities on a 1-10 scale, according to the stress they generated. Three categories of stress-inducers came to the forefront: lost time (like not being able to work in locations without wireless or planes without access), surprise (like lost luggage) and interruptions of daily routines (like altered sleeping schedules or a lack of healthy foods).…
The humanitarian power of big data
Big data can provide insightful solutions for businesses in many different industries. As a few recent projects show, big data might be able to benefit humanitarian life-saving strategies and bring peace to embattled parts of the world. As researchers and experts in many different fields are able to access and utilize big data from many different sources to inform their work, they are discovering ways in which analytics can create positive socio-cultural change.
International humanitarian advocate Patrick Meier recently wrote on Forbes about 'crisis maps,' tools for solving social problems with data. Innovations like crisis maps are fairly new, developed using open source big data insights. Real-time social communications converge with ethnographic and geographic research to provide insight during disastrous events.
"Crisis-mapping platforms display eyewitness reports submitted via e-mail, text message, and social media. The reports are then plotted on interactive maps, creating a geospatial record of events in real time," wrote Meier.…
Enterprise Big Data Analytics with Hortonworks and Datameer
Today, 94% of Hadoop users perform analytics on large volumes of data that were not possible before. How do they do it? Cool applications, that’s how.
You have seen various stats that indicate enterprises need better ways of making use of data but they bear repeating: The volume of business data worldwide, across all companies, doubles every 1.2 years, according to a study published by eBay in May, 2012. And market research firm IDC released a forecast showing the big data market may grow from $3.2 billion in 2010 to $16.9 billion in 2015. Clearly, enterprises need better ways of making use of all of this data, which contains innumerable insights for improving business processes and profitability.
Hortonworks partner Datameer, has a horizontal application for big data discovery that includes self-service data integration, analytics and visualization on top of Hadoop, including pre-built analytic applications.
While Datameer itself is a horizontal application for big data discovery that includes self-service data integration, analytics and visualization on top of Hadoop, Datameer takes it one step further and even offers pre-built analytic applications. Datameer’s Analytics App Market is the world’s first marketplace for buying and selling analytic applications that allows users to simply plug in their own data and see the final results visualized, without having to do the work of building the analysis.
The applications are downloaded with a single click, and range from broad, horizontal use cases that most any organization could utilize like email analytics or social media brand sentiment analysis to very specific use-case driven applications like Zendesk Forum analytics or JIRA ticket analyses. The best part is the marketplace is constantly growing as data scientists and subject matter experts from around the world create and contribute new applications for virtually any structured or unstructured data source.
Betting on Hadoop
Joe Nicholson, VP of marketing at Datameer, explains that the idea of Big Data analytics has exploded in the past 5 years. Business intelligence is not new, he said. What changed is the rise of so-called unstructured data. “Today, companies want to track things like customer paths taken through a website, email network usage, comments posted on websites or collaborative tools, or find the useful information hidden in millions and millions of tweets,” said Nicholson.
“There’s no way to do it all in any sort of timely fashion, or without breaking the bank, without first getting all of your data in Hadoop. So first and foremost comes the need to be able to get that data in yourself, without relying on IT. Then you want to point-and-click your way through your analysis and get instant feedback so you can analyze the same way you think. When you’ve built your analysis, that’s when you want to run it against your entire dataset. And finally, you want to visualize your results with just a few clicks. Import corporate logos, add text, make the report your own. We do all of that in Datameer, and we couldn’t do it if we hadn’t made this fundamental bet on Hadoop.”
Datameer partnered with Hortonworks back in 2011, and the two companies have been working together to accelerate the development and adoption of big data analytic solutions that leverage or extend the Apache Hadoop platform, and allow users to tap into the massive amounts of unstructured data.
The joint webinar conducted earlier this year, “Big Data Analytics: Is Your Elephant Enterprise Ready?” addressed critical project components such as data security, high availability, user training and use case development.
For more information on Datameer visit www.Datameer.com or @datameer
Big data’s potential for higher education
One of the many arenas where officials could benefit from big data deployment is higher education. With their large volumes of student information, including enrollment, academic and disciplinary records, universities have the datasets needed to benefit from a targeted analytics project.
Recent reports have suggested that administrators across the United States have become to realize the potential of analytics and have invested more resource into big data projects. Gartner's "Big Data, Bigger Opportunities: Investing in Information and Analytics" survey found that 42 percent of IT professionals across all sectors reported investing in big data projects or planning to do so in the coming year, reported Campus Technology. The report also identified higher education as field with great promise for big data application, citing the availability of various sources of data.
More data brings greater insights
Once universities begin investing more in Hadoop big data projects, they benefit from its many applications.…
Building better cars with big data
When car companies across the United States saw their funds quickly dwindle at the end of 2008, some predicted the end of the American auto industry. Several of the nation's largest automakers stood on the brink of bankruptcy and slashed jobs in order to avoid complete financial collapse. According to information presented by the White House, at the height of the automotive crisis, nearly 150,000 industry workers lost their jobs in a single quarter. When economic circumstances become dire and markets fluctuate, businesses need to pursue new methods of operation to weather the financial storm. In the face of economic turmoil, the Ford Motor Company turned to big data analytics to help it regain its footing in the global auto industry.
Integrating customer demands with the design process
Big data, along with the development of increasingly sophisticated sensor equipment, has allowed Ford engineers to more effectively analyze the performance of their automobiles and build vehicles based on a balance of quality and customer demands.…
Big data takes on the role of human resources
Since the recent recession hobbled the economy, Americans have struggled with unemployment. Bob Eisenbreis, a chief monetary economist with New York's Cumberland Advisers, said that although the unemployment rate is currently recognized as 7.7 percent, the true level is 14.3 percent when factoring in the underemployed, reported Forbes.
One of the hardest hit demographics in the recession has been young adults between 16 and 24. According to the Center for American Progress, the unemployment rate for that demographic stands at 16.2 percent. Furthermore, the organization found that experiencing an extended period of unemployment early in a career could have an adverse effect on an individual's earning power later in life. A prospective worker who has been unemployed for at least a period of six months stands to lose $22,000 over the course of the ensuing decade. The organization's analysis concluded that young Americans will lose approximately $20 billion in potential earnings in the next 10 years.…
Big data’s potential drives healthcare spending for epidemiology and genomes
Big data has shown enormous potential in the healthcare industry and enterprises are beginning to respond. According to the latest study released by TechNavio, the global big data market in the healthcare sector is expected to increase at a compound annual growth rate of 32.96 percent through 2015.
Researchers found that one of the major drivers behind the market growth has been increased access to data. The advent of electronic health records and the federal government's push to standardize their usage has resulted in more organizations obtaining the information needed to launch a successful data analytics program. The study also discovered that more medical professionals were deploying analytics tools to enhance their epidemiology research efforts and uncover new insights into the patterns and causes of various ailments.
More sophisticated tools lead to greater insights
The potential for data analytics to improve patient care and save lives has fostered excitement from many within the medical community.…
Big data takes to the high seas
Piracy off the coast of Somalia has grabbed a lot of headlines in recent years. Stories of ships being boarded by pirates on the Indian Ocean and the Gulf of Aden have captured the attention of people around the globe. While piracy may be a mere curiosity to some, it has very real ramifications. The Economist recently reported that the cost alone in ransom payments made to ensure the safety of captured sailors was estimated to be $53 million each year. The exact total cost of piracy to the world economy is still up for debate, but experts agree that it is significant. Oceans Beyond Piracy's analysis concluded that the annual cost of these crimes totaled $6 billion. That figure pales in comparison to the World Bank's estimates, however, which place the annual cost of piracy at $18 billion.
Somali government officials have grappled with the problem for years.…
Big Data Defined – Part Deux: Value Definition
A few weeks back we posted a definition of “big data”. There was definitely some internal conversation about the term and if this definition had captured what the term means. Sum finding: it is a loaded term. It means a lot of different things to a lot of different people.
When I first joined Hortonworks, I bought in to the three V’s (volume velocity and variety) definition of big data. It works for the most part, but is more a descriptor of the data. It explains the characteristics of the data. The definition is cold and lacks soul. Afterall, “big data” represents promise of “big” business value.
A “Value” Definition of Big Data
Last year, Shaun Connolly, Hortonworks VP of Corporate Strategy came up with this definition…
Big Data = Transactions + Interactions + Observations.
I gravitate to this because it outlines WHAT the data is, not just the characteristics. It points to areas that we should focus on as businesses. It lends to the value a bit more. Each of the three components are important.
- Transactions are pretty simple to understand. This is our ERP data. It is the data that we maintain and track in our OLTP systems. It can be any record of any system-to-system or human-to-system interaction. It can even be a human-to-human interaction as long as it is captured electronically. We use a lot of this data in our analytics today.
- Interactions are the points in time we relate with a system. It could be a tweet or a facebook post. It could be an electronic or paper customer satisfaction survey. Interactions are web logs and A/B tests. We have a lot of this data but typically no efficient way to understand or extract value from it.
- Observations are interesting because they represent a world of net new data sources that we once never thought of analyzing. It is data that was once thought of as low to medium value data or even exhaust data that was too bulky and just too expensive to store. This can be machine-generated data from sensors or web logs and clickstreams or even audio/video or largely unstructured content. Typically, we never even thought of this data before.
The Intersection Is Where Things Get Interesting
This “value” definition of big data gets interesting when you substitute the plus signs in Shaun’s definition with intersections…
Big Data = Transactions ∩ Interactions ∩ Observations.
With big data technology (one of these being Apache Hadoop) we can now efficiently store and process all of this data. We can refine observation data down to the salient details that may be interesting in the context of our EDW. But even more interesting we can ask these big data systems new questions. We can combine data across all these types and come up with new value for organizations. There is a world of data in our organizations that are used for an explicit purpose. When we start to combine things, the big data world gets really interesting.
If you’re using Hadoop to create value from your big data, why not check out our Hadoop Patterns of Use whitepaper and see how it can work for you.
Improving medical diagnosis, testing with big data
The healthcare industry is rife with patient information that could be leveraged with big data tools to provide more focused treatment and reduce operational expenses for medical facilities. The increased adoption rate of electronic health record systems has given more healthcare organizations the datasets needed to glean meaningful insights about their operations. According to a report issued by the Office of the National Coordinator for Health Information Technology, the number of acute-care hospitals using EHR systems tripled between 2019 and 2012, rising to 44 percent. Although hospital officials have only just begun to scratch the surface of the potential presented by Hadoop big data, many facilities have launched analytics programs that showcase the benefits of the technology.
Providing more accurate diagnoses
CIO.com recently outlined several aspects of the healthcare industry that could be immensely improved with big data tools. For instance, data analytics could help mitigate the risk resulting from "cookbook medicine" diagnosis methods.…
Big data’s potential for realtors
The real estate industry experienced a considerable rough patch over the past few years. The recent recession sapped many prospective home owners of their buying power, leading to fewer sales and plummeting prices. The real estate market has only just recently shown signs of improvement. A forecast released by the Urban Land Institute and Ernst & Young predicted the national average home price would rise 6 percent this year before leveling out at 5 percent in 2015, according to the National Mortgage Professional Magazine. In addition, the Christian Science Monitor reported that figures compiled by the National Association of Realtors showed that the median price of houses sold in March of this year had risen nearly 12 percent compared to the same period in 2012.
The real estate industry still has a long way to go, however. Inman News pointed out that there are several obstacles for realtors to overcome in order to increase their sales.…
Big data aids video game designers
As the Hadoop big data market continues to grow, more enterprises are finding new ways to take advantage of the technology's superior processing power. Regardless of their industry or sector, any organization that generates data can utilize these tools. The video game industry, for example, has long been an excellent candidate for big data implementation. According to the NDP Group's annual market study, video game sales in the United States totaled $14.8 billion in 2012, Financial Post reported.
Not only does the industry as a whole generate massive revenue, but the individual projects can require substantial investments from companies. The New York Times reported that one recent release, "BioShock Infinite," cost developers an estimated $100 million dollars. Analysts who spoke to the news outlet said marketing expenses to support the game could be expected to add another $100 million to total production costs.…