Hortonworks introduced support for Livy interpreter in Zeppelin with HDP 2.5. At that time, we integrated Livy 0.2 into Zeppelin which supports 4 types of Livy interpreters
We have received extensive feedback from our customers and the community about what they liked about Livy and the issues they experienced. In HDP 2.6, we have upgraded Livy to version 0.3 and have made a number of improvements in Livy and Zeppelin to deliver better user experiences for our customers. Specifically, these enhancements would prevent users from making errors by pre-selecting the default mode, improve productivity by restarting and recreating livy sessions and supporting functionality provided by Spark 2.0.
In this blog, I will highlight several changes and improvements we have made.
In HDP 2.5, we allowed users to choose the mode for Livy interpreter, although we encouraged them to use yarn-cluster mode. However, we got feedback from users that they experienced issues due to the incorrect mode specified. So in HDP 2.6, we have enforced the use of yarn-cluster mode which is also the default mode that Livy uses. The property `Livy.spark.master` has been removed from the Livy interpreter setting.
In HDP 2.5, we used Zeppelin’s shared mode for Livy interpreter. Shared mode requires that all the notes and users share the same Livy interpreter instance of Zeppelin. This makes it difficult to add new features to Livy interpreter. So in HDP 2.6, we changed Livy interpreter to scope based on user mode where each user’s Livy interpreter are in a different instance but in the same JVM. There are two benefits for scoped mode.
In HDP 2.5, Livy interpreter only supported Spark 1.6. However, in HDP 2.6, we support both Spark 1 and Spark 2. We can also have two Livy Servers deployed in HDP 2.6 – one for Spark 1.6 and another for Spark 2.x. The Livy servers are components of Spark 1 and Spark 2 respectively.
Although Zeppelin front end displays the most relevant exception messages, often there is need to check the YARN application log to get the details. In this scenario, unless you know the YARN app Id of your Livy session, it is hard to find the YARN app in YARN Resource Manager (RM) UI. In HDP 2.6, we display the YARN app id in Zeppelin and also show its web UI link when the property zeppelin.livy.displayAppInfo is set to true.
By default, Livy session expires after one hour. In HDP 2.5 after Livy session expires the user needs to restart the Livy interpreter. In HDP 2.6, we provided the ability to recreate the session automatically for the user and display a warning message in front-end as shown in the screenshot below.
Livy 0.3 supports cancellation of a statement. We have integrated this functionality into Zeppelin as well. Users can now click the cancel button to in order to terminate the running statement.
Livy 0.3 supports Python 3 that has been enabled in Zeppelin via %Livy.pyspark3. An additional configuration that user needs to specify is Livy.spark.yarn.appMasterEnv.PYSPARK3_PYTHON in Livy interpreter setting of Zeppelin
In HDP 2.6, we took a major step forward to improve the user experience of Livy interpreter in Zeppelin. But there are some known issues that we need to address in the next release. Please see below the list of candidates that we will be working to resolve in the next release. We encourage you to ask questions and create tickets in Zeppelin community at https://zeppelin.apache.org/