newsletter

Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

AVAILABLE NEWSLETTERS:

Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
January 22, 2014
prev slideNext slide

The Windows Explorer experience for HDFS

This guest post from Simon Elliston Ball, Head of Big Data at Red Gate and all round top bloke. 

Hadoop is a great place to keep a lot of data. The data-lake, the data-hub and the data platform;  it’s all about the data. So how do you manage that data? How do you get data in? How do you get results out? How do you get at the logs buried somewhere deep in HDFS?

At Red Gate we have been working on some query tools for Hadoop for a while and while testing we found ourselves endlessly typing hadoop fs. Getting data sets from our Windows desktops, to the cluster, or inspecting job output files was just taking too many steps. It should be as easy for us to access files on HDFS as files on my local drive. So we created HDFS Explorer, which works just like Windows Explorer, but connects to the WebHDFS APIs so we can browse files on our clusters.

HDFS Explorer helps if you’re shunting smaller data sets, or results to and from your desktop, but we also found it worked great for clearing up after all those test queries. Every job submission you send will leave a trail of meta-data, logs, job files, and output, which can quickly add up to a decent amount of disk. If you’re experimenting with a sandbox implementation this can be a real issue. On a proper cluster, even if disk space is not a problem, the mess left behind can make it hard to get to the right job’s diagnostics.

This video shows how to get up and running with HDFS Explorer and Hortonworks Sandbox so you can manage files and even clean up after yourself.

From its beginning as a humble little test tool, we’ve found HDFS Explorer has opened up the Hadoop File System, and made it much easier for us to implement proper data file management. This week we’re also happy to announce that we’ve added Kerberos support, making it ready for use on clusters in an Enterprise Authentication environment.

HDFS Explorer is available for free from the Red Gate Big Data site.

Download the Hortonworks Sandbox and get started with these great tutorials.

Tags:

Comments

Robert Vecchione says:

How do I browse HDFS with the WINDOWS version of HDP?

Philip H says:
Your comment is awaiting moderation.

The links for Red Gate’s Big Data site and the HDFS Explorer aren’t valid. Can’t seem to find a download on Red Gate’s site anywhere. Anyone know where I can find this tool?

Alex Mc says:
Your comment is awaiting moderation.

Is this product still available? I can’t see it on the Red Gate website.

Nir says:
Your comment is awaiting moderation.

where can i download this tool?

Eric Aldinger says:
Your comment is awaiting moderation.

This product is no longer available on the RedGate site. In fact that bigdata.red-gate.com subdomain redirects to RedGate.com.

Nigel Jones says:
Your comment is awaiting moderation.

It appears redgate’s hdfs explorer is no longer available for download. I’ve used it in the past and it was useful. Has anyone come across an alternative ?

Ki says:

What happened to HDFS Explorer? I can’t find it anywhere on Redgate’s site.

Peter says:

The url link are not working anymore! Can you update?

www.5axxw.com says:

i see,but it’not lastest!

Leave a Reply

Your email address will not be published. Required fields are marked *