There’s clearly a lot of hype [and confusion] in this emerging Big Data market, and it feels as if each new technology, as well as existing technologies, are pushing the meme of “all your data are belong to us”. It is great to see the wave of innovation occurring across the landscape of SQL, NoSQL, NewSQL, EDW, MPP DBMS, Data Marts, and Apache Hadoop (to name just a few), but enterprises and the market in general can use a healthy dose of clarity on just how to use and interconnect these various technologies in ways that benefit the business.

In my post entitled 7 Key Drivers for the Big Data Market, I asserted that the Big Data movement is not only about the classic world of transactions, but it factors in the new(er) worlds of interactions and observations. This new world brings with it a wide range of multi-structured data sources that are forcing a new way of looking at things.

In order to make sense of this emerging space, I’ve created two graphics designed to walk through a vision of a next-generation data architecture. At the highest level, I describe three broad areas of data processing and outline how these areas interconnect.

The three areas are:

1. Business Transactions & Interactions
2. Business Intelligence & Analytics
3. Big Data Refinery

The graphic below illustrates a vision for how these three types of systems can interconnect in ways aimed at deriving maximum value from all forms of data.

Enterprise IT has been connecting systems via classic ETL processing, as illustrated in Step 1 above, for many years in order to deliver structured and repeatable analysis. In this step, the business determines the questions to ask and IT collects and structures the data needed to answer those questions.

The “Big Data Refinery”, as highlighted in Step 2, is a new system capable of storing, aggregating, and transforming a wide range of multi-structured raw data sources into usable formats that help fuel new insights for the business. The Big Data Refinery provides a cost-effective platform for unlocking the potential value within data and discovering the business questions worth answering with this data. A popular example of big data refining is processing Web logs, clickstreams, social interactions, social feeds, and other user generated data sources into more accurate assessments of customer churn or more effective creation of personalized offers.

More interestingly, there are businesses deriving value from processing large video, audio, and image files. Retail stores, for example, are leveraging in-store video feeds to help them better understand how customers navigate the aisles as they find and purchase products. Retailers that provide optimized shopping paths and intelligent product placement within their stores are able to drive more revenue for the business. In this case, while the video files may be big in size, the refined output of the analysis is typically small in size but potentially big in value.

The Big Data Refinery platform provides fertile ground for new types of tools and data processing workloads to emerge in support of rich multi-level data refinement solutions.

With that as backdrop, Step 3 takes the model further by showing how the Big Data Refinery interacts with the systems powering Business Transactions & Interactions and Business Intelligence & Analytics. Interacting in this way opens up the ability for businesses to get a richer and more informed 360 ̊ view of customers, for example.

By directly integrating the Big Data Refinery with existing Business Intelligence & Analytics solutions that contain much of the transactional information for the business, companies can enhance their ability to more accurately understand the customer behaviors that lead to the transactions.

Moreover, systems focused on Business Transactions & Interactions can also benefit from connecting with the Big Data Refinery. Complex analytics and calculations of key parameters can be performed in the refinery and flow downstream to fuel runtime models powering business applications with the goal of more accurately targeting customers with the best and most relevant offers, for example.

Since the Big Data Refinery is great at retaining large volumes of data for long periods of time, the model is completed with the feedback loops illustrated in Steps 4 and 5. Retaining the past 10 years of historical “Black Friday” retail data, for example, can benefit the business, especially if it’s blended with other data sources such as 10 years of weather data accessed from a third party data provider. The point here is that the opportunities for creating value from multi-structured data sources available inside and outside the enterprise are virtually endless if you have a platform that can do it cost effectively and at scale.

Let me conclude by describing how the various data processing technologies fit within this next-generation data architecture.

In the graphic above, Apache Hadoop acts as the Big Data Refinery. It’s great at storing, aggregating, and transforming multi-structured data into more useful and valuable formats.

Apache Hive is a Hadoop-related component that fits within the Business Intelligence & Analytics category since it is commonly used for querying and analyzing data within Hadoop in a SQL-like manner. Apache Hadoop can also be integrated with other EDW, MPP, and NewSQL components such as Teradata, Aster Data, HP Vertica, IBM Netezza, EMC Greenplum, SAP Hana, Microsoft SQL Server PDW and many others.

Apache HBase is a Hadoop-related NoSQL Key/Value store that is commonly used for building highly responsive next-generation applications. Apache Hadoop can also be integrated with other SQL, NoSQL, and NewSQL technologies such as Oracle, MySQL, PostgreSQL, Microsoft SQL Server, IBM DB2, MongoDB, DynamoDB, MarkLogic, Riak, Redis, Neo4J, Terracotta, GemFire, SQLFire, VoltDB and many others.

Key Takeaway

A next-generation data architecture is emerging that connects the classic systems powering Business Transactions & Interactions and Business Intelligence & Analytics with Apache Hadoop, a “Big Data Refinery” capable of storing, aggregating, and transforming multi-structured raw data sources into usable formats that help fuel new insights for the business.

Enterprises that can maximize the value from all of their data (i.e. transactions, interactions, and observations) will put themselves in a position to drive more business, enhance productivity, or discover new and lucrative business opportunities.

~ Shaun Connolly

“What is Big Data?” and “What are the drivers behind the Big Data market?”

While most definitions of Big Data focus on the new forms of unstructured data flowing through businesses with new levels of “volume, velocity, variety, and complexity”, I tend to answer the question using a simple equation:

Big Data = Transactions + Interactions + Observations

The following graphic illustrates what I mean:

ERP, SCM, CRM, and transactional Web applications are classic examples of systems processing Transactions. Highly structured data in these systems is typically stored in SQL databases.

Interactions are about how people and things interact with each other or with your business. Web Logs, User Click Streams, Social Interactions & Feeds, and User-Generated Content are classic places to find Interaction data.

Observational data tends to come from the “Internet of Things”. Sensors for heat, motion, pressure and RFID and GPS chips within such things as mobile devices, ATM machines, and even aircraft engines provide just some examples of “things” that output Observation data.

With that basic definition of Big Data as background, let’s answer the question:

What are the 7 Key Drivers Behind the Big Data Market?

Business

1. Opportunity to enable innovative new business models
2. Potential for new insights that drive competitive advantage

Technical

1. Data collected and stored continues to grow exponentially
2. Data is increasingly everywhere and in many formats
3. Traditional solutions are failing under new requirements

Financial

1. Cost of data systems, as a percentage of IT spend, continues to grow
2. Cost advantages of commodity hardware & open source software

There’s a new generation of data management technologies, such as Apache Hadoop, that are providing an innovative and cost effective foundation for the emerging landscape of Big Data processing and analytics solutions. Needless to say, I’m excited to see how this market will mature and grow over the coming years.

Key Takeaway

Being able to dovetail the classic world of Transactions with the new(er) worlds of Interactions and Observations in ways that drives more business, enhances productivity, or discovers new and lucrative business opportunities is why Big Data is important.

One promise of Big Data is that companies who get good at collecting, aggregating, refining, analyzing, and maximizing the value derived from Transactions, Interactions, and Observations will put themselves in a position to answer such questions as:

What are the behaviors that lead to the transaction?

And even more interestingly:

How can I better encourage those behaviors and grow my business?

So ask yourself, what’s your Big Data strategy?

~ Shaun Connolly

HCatalog is a metadata and table management system that provides a consistent data model and schema for users of tools such as MapReduce, Hive and Pig. When you consider that there are often users accessing Hadoop clusters using different tools that independently don’t agree on schema, data types, how and where data is stored, etc., then you can understand the value of having a tool such as HCatalog.

In this video, Alan does a good job of not only explaining the role of HCatalog, but also laying out the future direction of the project. He talks about improving the integration with HBase, improving information lifecycle management and expanding the HCatalog data model to address the challenges of unstructured data.

If you would like to learn more about HCatalog or any of the Apache Hadoop projects, I strongly suggest that you attend Hadoop Summit next month. There will be a number of compelling sessions, including a presentation on HCatalog hosted by Alan Gates himself.

~ John Kreisa

Hortonworks and our co-host Yahoo! would like to thank the following companies for helping to make Hadoop Summit possible:

Platinum Sponsors

• Cisco Systems
• Datameer
• IBM
• Karmasphere
• MarkLogic
• Microsoft
• Savvis
• Splunk
• StackIQ
• Teradata Aster
• Vertica, An HP Company
• VMware

Gold Sponsors

• Cloudera
• Dropbox
• Intel
• Lucid Imagination
• MapR Technologies
• Pentaho
• Syncsort

Silver Sponsors

• Amazon Web Services
• Cloudwick Technologies
• Dataguise
• DataStax
• Dell
• Drawn to Scale
• Facebook
• Hadapt
• HStreaming
• Jive Software
• Lilien
• Mellanox Technologies
• NetApp
• Pervasive Big Data
• Quest Software
• Qubole
• SoftNet Solutions
• Super Micro Computer
• Tableau Software
• Talend
• Think Big Analytics
• Zettaset

Hadoop Summit would also like to thank its 2012 media sponsors: Datanami, DBTA, NoSQL Weekly and SiliconANGLE.

Registration for Hadoop Summit 2012 remains open however the conference is filling up fast. Don’t miss the opportunity to attend by registering today. For more information on the conference, please visit the Hadoop Summit session guide and schedule.

~ John Kreisa

If you’re not already doing so, I strongly suggest that you follow both Arun (@acmurthy) and Hortonworks (@hortonworks) on Twitter. You might also want to check out various presentations given by Arun and many of the engineers and executives at Hortonworks now available on Slideshare.net.

If you’re interested in Apache Hadoop training, please note every Developing Solutions using Apache Hadoop class between now and the end of June will include a lunch and learn session with Arun. Of course, you can also attend Hadoop Summit in June to meet the Hortonworks technical and executive team in person.

~ John Kreisa

Every talk of the day was excellent, from the use cases by Datasift, Mendeley and MusicMetric, to the talk by Francine Bennett of MastodonC on the CO2 footprint of different cloud computing infrastructures, including a live dashboard on the current CO2/hour of many cloud infrastructure sites.

In my discussions with attendees, I was impressed how broadly Hadoop is starting to be adopted in the U.K. There is adoption from “pure data” companies like Mendeley, DataSift, MusicMatch, Last.fm, as well as media companies and financial organisations. London is a centre of finance and data and as such, from a Hadoop perspective, it is a source of data waiting to be stored and mined.

The combination of enterprises, web companies and the infrastructure developers such as Canonical and -as of this week- Hortonworks, means that London and the South of England has a good opportunity to grow it’s big data community centred around Hadoop.

Steve Loughran
Hortonworks
Bristol, England

Boolean Data Type

Pig 0.10 introduces boolean data type as a first-class Pig data type. Users can use the keyword “boolean” anywhere where a data type is expected, such as load-as clause, type cast clause, etc.

Here are some sample use cases:

a = load ‘input’ as (a0:boolean, a1:tuple(a10:boolean, a11:int), a2);

b = foreach a generate a0, a1, (boolean)a2;

c = group b by a2; — group by a boolean field

When loading boolean data using PigStorage, Pig expects the text “true” (ignore case) for a true value, and “false” (ignore case) for a false value; while other values map to null. When storing boolean data using PigStorage, true value will emit text “true” and false value will emit text “false”.

Nested Cross/Foreach

You can use nested cross and nested foreach statements inside foreach nested plan in Pig 0.10. Here is one example:

C = cogroup user by uid, session by uid;
D = foreach C {
    crossed = cross user, session;
    filtered = filter crossed by user::region == session::region;
    result = foreach filtered generate processSession(user::age, user::gender, session::ip); -- processSession is a UDF
    generate result;
}

Note the maximum level of nested plan is 2, that is, the nested foreach statement cannot have a nest plan.

For more information, please refer to the Foreach section of the Pig documentation.

JRuby UDF

In addition to Python/JavaScript, in Pig 0.10, you can now use JRuby UDFs as well.

To write a JRuby UDF, you need to create a new JRuby class and extend PigUdf, and add your UDFs as methods of the new class. Here is one example:

require 'pigudf'
class Myudfs < PigUdf
    def concat *input
        input.inject(:+)
    end
end

There are two ways to define output schema for the UDF: annotation or schema function. Either is fine for defining the output schema in the previous sample:

class Myudfs < PigUdf
    outputSchema "word:chararray"
    def concat *input
        input.inject(:+)
    end
end

or,

schema function:
class Myudfs < PigUdf
    outputSchemaFunction :concatSchema
    def concat *input
        input.inject(:+)
    end
    def squareSchema input
        input
    end
end

You can also write algebraic and accumulative UDFs in JRuby, which is not yet the case for other scripting languages. For more information, please refer to the Pig documentation for Writing Ruby UDFs.

Hadoop 0.23 (a.k.a. Hadoop 2.0) Support

Pig 0.10.0 supports Hadoop 0.23.X. All unit and end-to-end tests passed with hadoop-0.23. To run Pig with hadoop-0.23, you need to recompile Pig with hadoopversion flag set to 23:

ant -Dhadoopversion=23

You also need to set up all of the environment variables necessary to run the hadoop -23 client, plus, point HADOOP_HOME to HADOOP_COMMON_HOME, and make sure $HADOOP_HOME/bin/hadoop exists.

Performance Improvements

Map Aggregation

Map aggregation will aggregate records before it sends them to combiner. It reduces the serializing/deserializing costs of using combiner by sending fewer records to the combiner. It is especially useful in a group-by statement with very few group keys. In our experiments, map aggregation reduces the runtime for a map task for a group by clause by up to 50%.

Map aggregation is turned off by default. To turn it on, set “pig.exec.mapPartAgg” property to true.

For more information about map aggregation, read the PIG-2228 JIRA.

Push Limit into Loader

Pig optimizes limit query by pushing limit automatically to the loader, thus requiring only a fraction of the entire input to be scanned.

Language Enhancements

Re-aliasing

In the Pig script, you can rename an alias, and refer to the new name:

A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float);

B = A;

DUMP B;

Limit/Sample by Expression

The Limit/Sample statement takes expression in addition to constant. For example:

a = load 'a.txt';

b = group a by all;

c = foreach b generate COUNT(*) as sum;

d = order a by $0;

e = limit d c.sum/100;

Default Split Destination

You can specify an “otherwise” destination for split statement. Split will automatically identify inputs that don’t belong to any of the other branches and direct those inputs to the “otherwise” destination:

split a into b if id > 3, c if id < 5, d otherwise;

TOMAP/TOTUPLE/TOBAG Syntax Support

You can compose a map/tuple/bag within a Pig script:

B = foreach A generate (name, age);  -- generate tuple

B = foreach A generate [name, age];  -- generate map

B = foreach A generate {name, age};  -- generate bag of single item tuples

Globbing in Register

Pig now supports globbing in register statements:

register lib/*.jar

UDF Enhancements

Improvements to PigStorage

We added a couple of options to PigStorage:

*-schema

This is for storing a .pig_schema along a data file when when using PigStorage. When loading data from PigStorage, Pig will check the existence of .pig_schema and use it automatically:

store a into 'output_dir' using PigStorage('\t', '-schema');

* -tagsource

PigStorage now adds a new column INPUT_FILE_NAME, which indicates the input file name of that input.

a = load 'input_dir' using PigStorage('\t', '-tagsource');

The first column of the output will be INPUT_FILE_NAME

Turn off the Write Ahead Log for HBaseStorage

You can now use the “-noWAL” option in HBaseStorage to turn off write ahead log while doing bulk loads into HBase:

STORE myalias INTO 'MyTable' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('mycolumnfamily:field1 mycolumnfamily:field2','-noWAL');

JsonLoader/JsonStorage

We added new pair of UDFs to the load/store Json format. Note JsonLoader does not auto detect the schema of your input data. You will still need to tell JsonLoader the schema of the data. Such as:

a = load 'input.json' using JsonLoader('a0:int,a1:{(a10:int,a11:chararray)},a2:(a20:double,a21:bytearray),a3:[chararray]');

However, if you are storing the data using JsonStorage, there will be a schema file stored along with the data. In this scenario, you don’t have to specify the schema for JsonLoader. JsonLoader will detect the schema file and use it.

Bloom Filters

Bloom filters are a common way to select a limited set of records before moving data for a join or other heavyweight operation. Pig includes two UDFs: BuildBloom to build a bloom filter and Bloom to use the bloom filter in a filter statement. At present, users will need to explicitly call both UDFs to get the full benefit of bloom filter. In the future, we will include them in the optimizer so that large join queries can use bloom filter automatically.

Please read the PIG-2328 JIRA for more information about bloom filters.

Implement UDF by Simulation

In the chain of EvalFunc -> Accumulator -> Algebraic, if you implement a more complex UDF (righthand side), you can use simulation to get a simpler UDF (lefthand side) for free. You can achieve this by using AlgebraicEvalFunc or AccumulatorEvalFunc. Check AlgebraicEvalFunc.html and AccumulatorEvalFunc.html for detail.

Other Improvements

Sparse Joins

Pig 0.10 introduces a new join type: ‘merge-sparse’. This is useful for cases when both joined tables are pre-sorted and indexed, and the right-hand table has few ( < 1% of its total) matching keys. Further detail on sparse joins is available in the Pig documentation.

Complete S3 Support

In Pig 0.10, every component of a Pig script can be in HDFS or Amazon Web Service’s S3. This includes the Pig script file, dependent jars, parameter files, macros, scripting UDFs, etc.

Kill Hadoop Job

If you kill a Pig job using Ctrl-C or “kill”, Pig will now kill all associated Hadoop jobs currently running. This is applicable to both grunt mode and non-interactive mode.

Conclusion

Remember to check out Pig 0.10 when you get a chance. Also, don’t forget to register for Hadoop Summit 2012. There will be some useful Pig presentations including my session with Thejas Nair, Pig Programming is More Fun: New Features in Pig.

~ Daniel Dai

A snapshot is a point-in-time image of the entire filesystem or a subtree of a filesystem. Some of the scenarios where snapshots are very useful:

1. Protection against user errors:  Admin sets up a process to take read-only (RO) snapshots periodically in a rolling manner so that there are always x number of RO snapshots on HDFS. If a user accidentally deletes a file, the file can be restored from the latest RO snapshot that contains the file.
2. Backup: Admin wants backup the entire file system, a subtree in the file system or just a file. Depending on the requirements, admin takes a read-only (henceforth referred to as RO) snapshot and uses this snapshot as the starting point of a full backup. Incremental backups are then taken by doing a diff between two snapshots.
3. Experimental/Test setups:  A user wants to test an application against the main dataset. Normally, without doing a full copy of the dataset, this is a very risky proposition because the test setup can corrupt/overwrite production data. Admin creates a read-write (henceforth referred to as RW) snapshot of the production dataset and assigns the RW snapshot to the user to be used for experiment. Changes done to the RW snapshot will not be reflected on the production dataset.
4. Disaster Recovery:  RO Snapshots can be used to create a consistent point in time image for replication and this can be copied over to remote site for Disaster Recovery.

High Level Requirements

1. Read-only (RO) snapshots: These are immutable copies of underlying elements of the file system.
2. Read-write (RW) snapshots: RW snaps can be modified by a user.
3. Support for taking snapshots of the entire namespace, or a subtree.
4. Support for a reasonable number of snapshots in a single namenode.
5. Snapshots should be easy to browse using standard commands and tools, and copying of data from a snapshot should work with standard Hadoop commands and API.

High Level approaches

We considered two options for snapshots.

Option #1: Both datanodes and namenode are aware of the snapshots and save state internally about the snapshots. Datanode is aware of the fact that some of the blocks are for the snapshot files.

Option #2: Only namenode is aware of the snapshot. Datanode is not aware of the fact that some of the blocks are owned by snapshots of the original file.

Option #2 is selected to keep the design simple. Additionally, taking snapshots is very fast with option #2. Datanode does not know anything about snapshots and is not aware of block ownership issues between root file system and snapshots. Keeping datanodes free from snapshot information simplifies the design immensely by eliminating the need for distributed co-ordination from the design of the snapshots by restricting the changes to namenode only.

Creating and Deleting Snapshots

A key requirement is to ensure that it is very easy to create and delete snapshots. Snapshot creation and deletion is an admin-only capability. To create a snapshot, one specifies a  snapshot name, a path to the root of the subtree whose snapshot is to be taken, and whether or not the snapshot is read-only or a read-write. Deleting snapshot requires just a snapshot name. A command to list all the snaps in the filesystem will be provided.

Accessing Directories and Files in a Snapshot

Snapshots can be referenced with regular HDFS path names with a reserved string .snapshot_<name>:

 hdfs://host:port/pathOfSnapshot/.snapshot_<name>/restOfPathInSnapshot

This has the benefit that snapshots can be referenced with all existing Hadoop commands and APIs that take a pathname by adding a reserved snapshot string to the pathname.

Examples:  Consider a directory structure of /a/b/c/foo.txt. Admin has created a snapshot hdfs1 at /a/b. To access data related to snapshot hdfs1, some examples of the commands would be:

hadoop dfs -ls /a/b/.snapshot_hdfs1/c/foo.txt 

To copy file from /temp/foo/foo1.txt in snapshot branch to /fooBar would be,

hadoop dfs -cp /a/b/.snapshot_hdfs1/c/foo.txt /foobar/.

Some caveats for RO snapshots include the fact that RO snapshot is immutable. So, operations such as creating a new file, deleting a file, creating a new directory, renaming a file or directory will fail when executed on the snapshot branch. 

Conclusion

Snapshots are a very useful feature to have in a mature filesystem. This is a work in progress and we have a functional prototype implemented. The first version of this feature will support RO snapshots only. The support for RW snapshots will be added in the subsequent releases. There are several features that can be incorporated into snapshots, such as time to live for snapshots with auto deletion, schedule based creation of snapshots, marking specific directories as snapshot-worthy, quota based restriction on space used by RW snapshots and delegation of authority for creating/deleting snapshots at specific locations to users etc.

To track the development of snapshots feature in HDFS, please follow the jira HDFS-2802.

~ Hari Mankude

In this video, Matt provides an overview of Hortonworks Data Platform (HDP), including a summary of the Apache Hadoop components included in the distribution and the testing processes involved in the release process. Matt also provides an overview of Apache Ambari, an open source project that is adding monitoring and management capabilities to Apache Hadoop.

If you are interested in learning more about Hortonworks technology, please visit the HDP page or attend one of the live or on-demand Webinars. Even better, consider attending the Hadoop Summit conference in June, where you will have a chance to learn from many of the core Apache Hadoop developers and other community leaders. You should also consider attending a Hortonworks training class, either in conjunction with Hadoop Summit or at another time and location more convenient to your schedule.

~John Kreisa

Citrix announced today that they were reaffirming their commitment to open source by working with the Apache Software Foundation to make CloudStack 3 an Apache project, released under Apache Software License 2.0. This is yet further acknowledgement that Apache is the logical home for open source projects that are transforming the enterprise software industry. As a Gold Sponsor of the ASF and major contributor to Apache projects, Hortonworks is pleased that leading vendors such as Citrix are recognizing the value that Apache can provide in terms of accelerating development and innovation and driving adoption as the preferred destination for enterprise-class open source software.

Today’s announcement also highlights the great synergies between CloudStack and Apache Hadoop. As the first cloud platform in the industry to join the ASF, CloudStack becomes the logical cloud choice for organizations that prefer an open source option for their cloud and big data infrastructure. Hortonworks is excited to work with the CloudStack project team to identify opportunities where Hadoop components can be used to back Cloud APIs and also where Cloud APIs can be used to deploy Hadoop.

Today marks a win for the Apache Software Foundation, CloudStack, Apache Hadoop and Hortonworks. The leading open source cloud community with more than 30,000 active members today joins forces with the Apache Software Foundation. This announcement is also further validation that Apache projects are the right place for developing software for enterprise grade platforms to meet the demanding cloud and Big Data needs of the enterprise.

~E14