Apache Pig

A scripting platform for processing and analyzing large data sets

Apache™ Pig allows you to write complex MapReduce transformations using a simple scripting language. Pig Latin (the language) defines a set of transformations on a data set such as aggregate, join and sort.  Pig translates the Pig Latin script into MapReduce so that it can be executed within Hadoop®. Pig Latin is sometimes extended using UDFs (User Defined Functions), which the user can write in Java or a scripting language and then call directly from the Pig Latin.

What Pig Does

Pig was designed for performing a long series of data operations, making it ideal for three categories of Big Data jobs:

  • Extract-transform-load (ETL) data pipelines,
  • Research on raw data, and
  • Iterative data processing.

Whatever the use case, Pig will be:

  • Extensible. Pig users can create custom functions to meet their particular processing requirements.
  • Easy to program. Complex tasks involving interrelated data transformations can be simplified and encoded as data flow sequences. Pig programs accomplish huge tasks, but they are easy to write and maintain.
  • Self-optimizing. The system automatically optimizes execution of Pig jobs, so the user can focus on semantics.

How Pig Works

Pig runs on Hadoop and makes use of MapReduce and the Hadoop Distributed File System (HDFS). The language for the platform is called Pig Latin, which abstracts from the Java MapReduce idiom into a form similar to SQL. Pig Latin is a flow language whereas SQL is a declarative language. SQL is great for asking a question of your data, while Pig Latin allows you to write a data flow that describes how your data will be transformed.  Since Pig Latin scripts can be graphs (instead of requiring a single output) it is possible to build complex data flows involving multiple inputs, transforms, and outputs. Users can extend Pig Latin by writing their own functions, using Java, Python, Ruby, or other scripting languages.

The user can run Pig in two modes:

  • Local Mode. With access to a single machine, all files are installed and run using a local host and file system.
  • MapReduce Mode. This is the default mode, which requires access to a Hadoop cluster.

The user can run Pig in either mode using the “pig” command or the “java” command.

Try these Tutorials

Apache Top-Level Project Since
September 2010
Hortonworks Committers
5
Project Page

Try Pig with Sandbox

Hortonworks Sandbox is a self-contained virtual machine with HDP running alongside a set of hands-on, step-by-step Hadoop tutorials.

Get Sandbox
More posts on:
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.
Contact Us
Hortonworks provides enterprise-grade support, services and training. Discuss how to leverage Hadoop in your business with our sales team.
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.