cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

closeClose button

Hortonworks Sandbox Guide

Introduction

Welcome to the Hortonworks Sandbox! Look at the attached sections for sandbox documentation.

Outline

  1. Sandbox Docs – HDP 2.6
  2. Sandbox Port Forwards – HDP 2.6

Sandbox Docs - HDP 2.6

Outline

Release Notes

Apr 2017

  • Md5 VMware Virtual Appliance – 50aab8be7ef25418d475000045abe571
  • Md5 Virtualbox Virtual Appliance – 9cb91797dc2a53bf3799007c5c80770e
  • Md5 Docker – f1663bbff04721c3b1a32a39fef676e3
  • HDP Stack and Ambari: The Sandbox uses the following versions of Ambari and HDP stack. Please use the following release note links provided to view Ambari and HDP stack specific information.

Behavior Changes

  • New splash page
    • Removed Falcon
    • Added Workflow Manager

Known Issues

  • Unable to stop/restart Oozie service using ambari
    • Works from the command line
  • Atlas Schema Tab Doesn’t Appear, So Not Able to Assign Specific Hive Columns an Tag
  • HBase RPC Bindings not configured out of box

Limitations

This is a list of common limitations along with their workarounds.

  • RMP-3586 – Due to dependency of the underlying OS and Virtual machine application, the following may occur when suspending the virtual machine:
    • Region Server service for HBase may be stopped when returning back from suspended state. It will need to be restarted.
    • Ambari Metrics may be stopped when returning back from suspended state since it now uses an embedded HBase.
    • Workaround: Avoid having to suspend your virtual machine.

System Informaiton

Operating System and Java versions that the Sandbox has installed.

  • OS Version (docker container)
    • CentOS release 6.8 (Final)
    • Java Version (docker container)
    • openjdk version “1.8.0_111”
    • OpenJDK Runtime Environment (build 1.8.0_111-b15)
    • OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)
    • Updated from previous version
  • OS Version (Hosting Virtual Machine)
    • CentOS Linux release 7.2.1511 (Core)

Image File Sizes:

  • VMware – 10.75 GB
  • Virtualbox – 10.4 GB
  • Docker – 12 GB

Databases Used

These are a list of databases used within Sandbox along with the corresponding HDP components that use them.

  • Ambari: postgres
  • Hive Metastore : Mysql
  • Ranger: Mysql
  • Oozie: derby (embedded)

HDP Supported Components Not Installed

These components are offered by the Hortonworks distribution, but not included in the Sandbox.

  • Apache Accumulo
  • Apache Mahout
  • Hue

Newly Added HDP Supported Packages

These are packages that have recently been included into the Sandbox for this release.

  • Workflow Manager

HDP Services Started Automatically on Startup

When the virtual machine is booted up, the following services are started. If not specified, assume all are java processes. The users that launch the process are the corresponding names of the component. The processes are listed with their main class.

  • Ambari
    • AmbariServer – org.apache.ambari.server.controller.AmbariServer run as root user
  • Ambari Agent (non java process)
  • Flume
    • Application – org.apache.flume.node.Application
  • HDFS
    • Portmap – org.apache.hadoop.portmap.Portmap
    • NameNode – org.apache.hadoop.hdfs.server.namenode.NameNode
    • DataNode – org.apache.hadoop.hdfs.server.datanode.DataNode
  • Nfs
    • Portmap – Unlike the other processes that are launched by hdfs user, these are run as root user.
    • The nfs process doesn’t show up as a name for jps output
  • HIVE
    • RunJar – webhcat – org.apache.hadoop.util.RunJar Run as hcat user
    • RunJar – metastore – org.apache.hadoop.util.RunJar
    • RunJar – hiveserver2 – org.apache.hadoop.util.RunJar
  • Mapreduce
    • JobHistoryServer – org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
    • mapred is the user used to launch this process
  • Oozie
    • Bootstrap – org.apache.catalina.startup.Bootstrap
  • Ranger
    • UnixAuthenticationService – org.apache.ranger.authentication.UnixAuthenticationService Run as root user
    • EmbededServer – org.apache.ranger.server.tomcat.EmbeddedServer
  • Spark
    • HistoryServer – org.apache.spark.deploy.history.HistoryServer
  • YARN
    • ApplicationHistoryServer – org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
    • ResourceManager – org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
    • NodeManager – org.apache.hadoop.yarn.server.nodemanager.NodeManager
  • Zookeeper
    • QuorumPeerMain – org.apache.zookeeper.server.quorum.QuorumPeerMain
  • Zeppelin
    • ZeppelinServer – org.apache.zeppelin.server.ZeppelinServer

HDP Services Not Started Automatically on Startup

Because of the limited resources avaialble in the sandbox virtual machine environment, the following services are in maintenance mode and will not automatically start. To fully use these services, you must allocate more memory to the sandbox virtual machine. If you want these services to automatically start, turn off maintenance mode. The processes are listed with their main class.

  • Ambari Infra
  • Ambari Metrics
  • Atlas
    • Main – org.apache.atlas.Main
  • HDFS
    • SecondaryNameNode – org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
    • Since on a single node, secondary namenode is not needed, it is not started.
  • Falcon
    • Main – org.apache.falcon.Main
  • HBase
    • HRegionServer – org.apache.hadoop.hbase.regionserver.HRegionServer
    • HMaster – org.apache.hadoop.hbase.master.HMaster
  • Kafka
    • Kafka – kafka.Kafka
  • Knox
    • gateway.jar – /usr/hdp/current/knox-server/bin/gateway.jar
    • ldap.jar – /usr/hdp/current/knox-server/bin/ldap.jar This process is a mini ldap server
  • Spark
    • Livy server run as livy
    • Thrift Server – org.apache.spark.deploy.SparkSubmit run as hive user
  • Spark2
    • Livy server run as livy
    • Thrift server – org.apache.spark.deploy.SparkSubmit run as hive user
  • Storm
    • supervisor – backtype.storm.daemon.supervisor
    • nimbus – backtype.storm.daemon.nimbus
    • logviewer – backtype.storm.daemon.logviewer
    • core – backtype.storm.ui.core
    • drpc – backtype.storm.daemon.drpc

Further Reading


Sandbox Port Forwards - HDP 2.6

Introduction

Listed below are the ports that the HDP Sandbox forwards by default, and what software or purpose each port corresponds to.

Note: Last updated for HDP Sandbox 2.6.0.1.

Port Forwards

1111 -> 111 -- NFS gateway
2049 -> 2049 -- NFS gateway
2181 -> 2181 -- Zookeeper
3000 -> 3000 -- Grafana
4040 -> 4040 -- Spark
4200 -> 4200 -- Ambari Shell
4242 -> 4242 -- NFS gateway
4557 -> 4557 -- NiFi DistributedMapCacheServer
6080 -> 6080 -- Ranger
6188 -> 6188 -- Ambari Metrics Timeline Server
8000 -> 8000 -- Storm Logviewer
8005 -> 8005 -- Sqoop / Common Tomcat port
8020 -> 8020 -- HDFS
8032 -> 8032 -- Yarn ResourceManager
8040 -> 8040 -- NodeManager
8042 -> 8042 -- NodeManager
8050 -> 8050 -- Yarn ResourceManager
8080 -> 8080 -- Ambari
8082 -> 8082 -- Namenode UI
8086 -> 8086 -- Namenode UI
8088 -> 8088 -- Yarn ResourceManager
8090 -> 8090 -- Namenode UI
8091 -> 8091 -- Namenode UI
8188 -> 8188 -- YarnATS
8443 -> 8443 -- Knox
8744 -> 8744 -- StormUI
8765 -> 8765 -- Phoenix
8886 -> 8886 -- Ambari Infra
8888 -> 8888 -- Tutorials splash page
8889 -> 8889 -- Jupyter
8983 -> 8983 -- SolrAdmin
8993 -> 8993 -- Solr
9000 -> 9000 -- HST (Smartsense)
9090 -> 9090 -- NiFi
9090 -> 9091 -- NiFi SSL
9995 -> 9995 -- Zeppelin
9996 -> 9996 -- Zeppelin
10015 -> 10015 -- Spark
10016 -> 10016 -- Spark
10000 -> 10000 -- HiveServer2
10001 -> 10001 -- HiveServer2Http
10500 -> 10500 -- HiveServer2v2
10502 -> 10502 -- HiveServer2 Interactive UI
11000 -> 11000 -- Oozie
15000 -> 15000 -- Falcon
15002 -> 15002 -- Hive LLAP
16000 -> 16000 -- HBase Master
16010 -> 16010 -- HBase Master Info
16020 -> 16020 -- HBase Regionserver
16030 -> 16030 -- HBase Regionserver Info
18080 -> 18080 -- SparkHistoryServer
18081 -> 18081 -- Spark2 History Server
19888 -> 19888 -- JobHistory
21000 -> 21000 -- Atlas
33553 -> 33553 -- Hive LLAP
39419 -> 39419 -- Hive LLAP
42111 -> 42111 -- NFS
50070 -> 50070 -- Webhdfs
50075 -> 50075 -- Datanode
50079 -> 50079 -- NFS gateway
50095 -> 50095 -- Accumulo
50111 -> 50111 -- WebHcat
61888 -> 61888 -- LogsearchUI

2222 -> 22 -- Sandbox container SSH
15500 -> 15500 -- Port for custom use
15501 -> 15501 -- Port for custom use
15502 -> 15502 -- Port for custom use
15503 -> 15503 -- Port for custom use
15504 -> 15504 -- Port for custom use
15505 -> 15505 -- Port for custom use