Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
March 08, 2017
prev slideNext slide

Autonomous Vehicles Silicon Valley 2017 – The Future of Transportation will drive huge growth in data

As I sat in the audience of the 2017 Autonomous Vehicle Silicon Valley conference this week in Santa Clara, listening to luminaries present their visions for an Autonomous Vehicle future, I was struck by the change sweeping across the automotive industry. This was definitely not the same industry I have worked within over the last 25 years of my career. While individual speakers held differing views on exactly when this autonomous world would be upon us, three points became abundantly clear:

  1. Autonomous vehicles are redefining the automotive industry ecosystem as we know it.
  2. The very environments in which we will live in the future will be forever changed.
  3. The shift to autonomy will be accompanied by a sea of new data, requiring enterprises to reexamine the ways they will manage information in the future.


Massive Industry Restructuring

If you harbored any doubts that the Connected Vehicle is driving massive industry restructuring, you would need only check the list of speakers on conference agenda. While there were just two presentations by traditional automotive OEMs, the speaker roster was dominated by “new-age” transportation mobility providers such as Lift, Uber and Zipcar, in addition to a wide-range of suppliers of autonomous enabling technologies such as Head-Units, Lidar, Radar, Cameras, OTA Updates, Semiconductors and many others.


Should cities actually transform as envisioned at the conference, the very experience of living in a city will be unrecognizable from what it is today. Uber, Lyft, Zipcar and various government agencies each presented visions of future cities liberated from the congestion rooted in today’s privately owned vehicles, with land from today’s parking spaces and parking lots reclaimed by leisure-friendly outdoor cafes and tranquil parks.

Coming Soon

Further, it appears that these changes may begin very quickly indeed, with Lyft projecting a fully autonomous fleet in major markets within 5 years and further predicting that private car ownership will all-but-end within major U.S. cities by 2025.


Upon completion of the conference, I sat down and reconsidered the conference through my eyes as a data practitioner. Two conclusions stood out in my mind. First, as widely acknowledged at the conference, the autonomous vehicle will usher in a veritable avalanche of new data. Second, and more pointedly, organizations within the autonomous ecosystem must quickly formulate new data strategies that are commensurate with this challenge.

Data Volume and Variety

Consider for a moment both the data volume and variety that will be generated across the autonomous vehicle value chain. Two examples come to mind:

  • First, when developing autonomous driving capabilities, engineers collect and store huge quantities (up to 4 TB per vehicle per day) of vehicle test data including radar, LIDAR, video and GPS. This stored data (or “data-at-rest”) is then mined to develop autonomous driving machine learning algorithms.
  • Second, when connected and autonomous vehicles are actually operated in the real world, “real-time”, IOT-based data (or “data-in-motion”) flows into and out of the vehicle to support a wide range of commercial, entertainment, service and safety functions.

Consequently, providers of connected and autonomous systems must look beyond traditional relational database management (RDBMS) technologies toward modern data platforms able to accommodate the huge volume and variety of both data-in-motion and data-at-rest that will be generated by future connected and autonomous vehicle processes.


With Big Data impacting so many connected and autonomous vehicle processes, Hortonworks customers often ask me where to begin. To this, I usually offer three suggestions:

  1. Identify a connected vehicle use case that will provide immediate value to your organization.
  2. Be sure to identify and join forces with business process owners who can serve as champions for the project moving forward.
  3. Finally, I recommend assessing the state of the data required for the initiative. How accessible is it? A great initial step involves creating a “Data Lake” that will support future connected vehicle transformation initiatives. Without your ecosystem data under management, it’s difficult to move on.


In short, the race toward connected and autonomous vehicles is quickly accelerating, impacting both the automotive industry value chain and broader society in ways unimaginable just a few years ago. Concurrently, this will unleash a flood of new data, of unprecedented volume and variety. Industry leaders, recognizing this, are already building and evolving their Big Data foundations – so don’t wait, the race is on!


Gourav Ghoshal says:

If a mechanical engineer wish to be a part of this race, by undertaking graduate studies in United States, which major do you suggest he should opt for – Automotive Engineering or Industrial Engineering?

Aaron N Smith says:
Your comment is awaiting moderation.

LiDAR for automotive Market is estimated to be USD 735.0 million in 2025, growing at a CAGR of 28.32%, during the forecast period.

Read More:

Leave a Reply

Your email address will not be published. Required fields are marked *