Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics, offering information and knowledge of the Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

closeClose button
Hortonworks Customer
The MITRE Corporation Transforms Aviation

The MITRE Corporation® is a not-for-profit company that operates multiple federally funded research and development centers (FFRDCs). One of these FFRDCs is the Center for Advanced Aviation System Development (CAASD), which serves the public interest by advancing the safety, security, effectiveness, and efficiency of aerospace in the United States and around the world.

Transforming the National Airspace System Through the NextGen Program

CAASD has been working with the US Federal Aviation Administration (FAA) since 1958 to provide the safest, most efficient aviation system in the world. CAASD provides the FAA with advanced technical capabilities in systems engineering, operational analysis, safety analysis, and prototyping coupled with in-depth domain knowledge in air traffic management and airspace user operations relevant to the National Airspace System (NAS).

In fulfilling its mission, CAASD offers specialized data analytics, simulation and computer modeling capabilities to evaluate a broad spectrum of NAS systems and operations. CAASD researches the NAS’ complex system of systems to provide data-informed answers to fundamental questions that the FAA asks continuously, including:

  • How can the NAS be more efficient?

  • How can the NAS be safer?

Answering those efficiency and safety questions requires that the CAASD team re-create historical operational scenarios to detect and analyze key efficiency and safety events, such as significant traffic bottlenecks and aircraft operations that are outside normal flight procedures (e.g., speeds, altitudes). To do that, CAASD continuously ingests, stores and analyzes massive amounts of detailed flight data from a variety of sources and enriches it with other data, such as:

  • Pilot and air traffic controller voice recordings

  • Weather data

  • Terrain maps

  • Air traffic management system data, and

  • Aircraft schedules and flight plans

Altogether, CAASD has over a petabyte of data in its Hortonworks Data Platform (HDP™) clusters, deployed both on premises and in the cloud.

Airline Participation Required Strong Data Governance

Of course, the CAASD team knew that the quality of its analysis would be directly correlated to the amount, breadth, and quality of the source data that it could analyze. As part of FAA safety data-sharing initiatives, CAASD relied on airline flight data to aid in risk analysis for joint FAA-Industry risk mitigation.

From the beginning, it was apparent that strong data governance and security would be paramount to reassuring carriers that their data would be stripped of any data elements that could tie it back to the particular airline that provided it. With the required data security processes and governance features in place, airlines agreed to provide CAASD with sensitive and proprietary data. MITRE is the home to collect that data and perform safety and efficiency-based studies that benefit the aviation community and flying public.

MITRE’s HDP Solution: One Data Lake, Governed By One Set of Policies & Centrally Operated with Apache Ambari

The CAASD team migrated its existing Hadoop cluster onto Hortonworks Data Platform for two main reasons. It wanted to avoid vendor lock-in, and also wanted to get as close to the core technology as possible, so that it would be able to adopt new platform innovation in response to its changing needs.

Before, CAASD had to maintain two different clusters: one dedicated to MapReduce and another to HBase. HDP’s centralized YARN-based architecture allowed the group to manage cluster resources between HBase and MapReduce jobs.

Apache Ambari provided additional value as part of HDP. As a MITRE data architect put it, “[Before moving to HDP] we found ourselves having to write custom code to fit our emerging use cases into the technology.” Cluster setup was a manual, home grown process. With Ambari, CAASD has a platform that lowers operational barriers to growing its cluster. Ambari has eased the burden on System Administrators, giving them more agility.

Apache Spark and Apache Storm hold promise for future CAASD data workloads. CAASD is actively transitioning production MapReduce workflows into Spark to improve performance. Spark is also being used to prototype more advanced analysis using statistics and machine learning.

The Results: Improved Safety, Effectiveness and Compliance

The CAASD team uses the system to look at all the flight data holistically and enrich it with additional external data.


With more than a petabyte of data in its HDP data lake, CAASD has created derived data products. With flight data, combined with data from hundreds of different surveillance sensors, the team can now create logical “flight stories” for any growing number of specific flights. The derived data products are continually stored to enable historical analyses; the archive of flight stories currently spans more than five years.

That analysis could include simulations of terrain avoidance. It can be used to identify safety hot spots or to model traffic flows and densities. Through analysis, CAASD can detect and classify certain types of flight operations such as missed approaches or rejected takeoffs.

All of this visibility helps the FAA to identify systematic risks across the NAS and develop mitigation measures with airspace users, such as controller training improvements or changes to operational procedures.

Regulatory Efficiency

Many operational rules and regulations were enacted years ago, before more recent technological innovations and the ability to harness Big Data analytics.

Because of this, the CAASD team is working with the FAA to look for opportunities to improve regulatory flexibility without any increase in air traveler risk. Simply put, the team wants to identify regulations that increase cost and inconvenience, without actually making anybody safer. Making changes to those areas promises to improve the efficiency of the entire system—not to mention the cost and convenience of air travel.

An early win in this area has to do with “separation standards” (rules governing how far apart aircraft need to be from one and other.) Based on analysis on data in HDP, the FAA has altered some of its separation standards and improved efficiency. By FAA estimates, enacting such changes at a single large airport is translating into nearly $20M in saved fuel and a reduction of nearly one-and-a-half years of cumulative passenger taxi and departure times per year.

Next Steps

CAASD is actively growing its data in multiple dimensions. Highly dense weather data is continually being transformed within HDP to help enable real-time design feedback for more efficient flight procedures. In addition, CAASD is expanding its Hadoop data lake to include operations from around the globe to help validate and tune the next generation of airborne collision avoidance systems. These systems will not only increase safety for operations today, but will also help pave the way for safe integration of new entrants into the NAS, including Unmanned Aircraft Systems (UAS).

Reference herein to any specific commercial entity by trade name, trademark, or otherwise, does not constitute or imply its endorsement, recommendation, or favoring by The MITRE Corporation (MITRE).