The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

Spark Forum

Spark 1.0.1 Tech preview available

  • Author
    Replies
  • #59279
    Michael Moss
    Participant

    Hi,

    I went through the preview instructions and everything worked great. For those who would like to write a java/scala client, which spark-core version should we use via maven? I couldn’t find the jars in the Hortonworks maven repo, is one there? I was getting serialization errors with some of the spark classes when using “org.apache.spark” % “spark-streaming_2.10” % “1.0.2”. Should I use 1.0.1 from apache?

    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:0 failed 4 times, most recent failure: Exception failure in TID 3 on host ip-172-31-128-8.ec2.internal: java.io.InvalidClassException: org.apache.spark.rdd.RDD; local class incompatible: stream classdesc serialVersionUID = -6766554341038829528, local class serialVersionUID = 385418487991259089

    Best,

    -Mike

    #59384
    Vinay Shukla
    Moderator

    Mike,

    Try using 1.0.1 version of the spark-core since that’s what the TP is build with. When we revise the TP we will make sure to publish the spark-core jar to the HWRK repo.

    Please let me know how you make out.

    Thanks,
    Vinay

    #59386
    Gary Chia
    Participant

    hi

    got this error…..please kindly assist

    ./bin/spark-submit –class org.apache.spark.examples.SparkPi –master yarn-cluster –num-executors 3 –driver-memory 512m –executor-memory 512m –executor-cores 1 lib/spark-examples*.jar 10
    14/08/28 16:12:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
    14/08/28 16:12:41 INFO impl.TimelineClientImpl: Timeline service address: http://HDOP-M.AGT:8188/ws/v1/timeline/
    14/08/28 16:12:41 INFO client.RMProxy: Connecting to ResourceManager at HDOP-M.AGT/10.193.1.71:8050
    14/08/28 16:12:41 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 5
    14/08/28 16:12:41 INFO yarn.Client: Queue info … queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,
    queueApplicationCount = 0, queueChildQueueCount = 0
    14/08/28 16:12:41 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 13824
    14/08/28 16:12:41 INFO yarn.Client: Preparing Local resources
    14/08/28 16:12:41 WARN hdfs.BlockReaderLocal: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
    Exception in thread “main” org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode=”/user”:hdfs:hdfs:drwxr-xr-x
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5515)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5471)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3614)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3584)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3558)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:760)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)

    #59389
    Daryl
    Participant

    I have the same ‘Permission Denied’ exception as Gary. Installed this version of Spark using the steps in the given url on the Master node in a working Ambari HDP 2.1.3 cluster.

    #59393
    Daryl
    Participant

    I figured the permissions error out, using this link: http://blog.spryinc.com/2013/06/hdfs-permissions-overcoming-permission.html
    You need to add user ‘root’ to group ‘hdfs’.
    groupadd hdfs
    (This will probably return a notice that the group already exists)
    usermod -a -G supergroup root

    Now it runs and returns a link. Unfortunately, when I click the ‘logs’ link it forwards me to a non existing location.

    #59394
    Daryl
    Participant

    usermod -a -G supergroup root
    should be:
    usermod -a -G hdfs root

    Sorry for spamming, but I can’t seem to edit existing posts.

    #79377
    Jitu kumar
    Participant

    15/10/01 05:07:01 ERROR SparkContext: Error initializing SparkContext.
    org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode=”/user”:hdfs:hdfs:drwxr-xr-x
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:238)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:179)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6515)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6497)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6449)

    Hi, I am facing this issue during spark installation version 1.4. on Hadoop 2.2.0

    #79378
    Jitu kumar
    Participant

    Thanks in advance

    #79381
    Truong Nguyen
    Participant

    Thanks Mod very much

The forum ‘Spark’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.