Home Forums Pig jython "processing new jar" each time

This topic contains 5 replies, has 3 voices, and was last updated by  tedr 1 year, 4 months ago.

  • Creator
    Topic
  • #24680

    When using a jython script, the Jython interpreter has to do the “processing new jar” each time.
    This must be cached.

Viewing 5 replies - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #25506

    tedr
    Member

    Hi Vladislav,

    Ok, then what you are asking for is not help with using it, but pointing out something that we could program for more efficient running of Pig scripts, i.e. a feature request/improvement. After making sure that it is not already requested I’ll add this to the database.

    Thanks,
    Ted.

    Collapse
    #25421

    Hi,

    I never said there were errors in the trace, just that the jar are processed each time pig is started.
    There is jython caching mechanism to use.
    It could make pig script using fonctions requiring processing jar much faster.

    Collapse
    #25420

    tedr
    Member

    Hi Vladislav,

    I can see no errors in in what you have posted, are you saying that the script doesn’t work or what?

    Thanks,
    Ted.

    Collapse
    #25333

    Hi,

    Imagine a simple job using a simple jython script.
    myfuncs.py
    @outputSchema(“word:chararray”)
    def concat(str):
    return str+str

    Register ‘myfuncs.py’ using jython as myfuncs;

    Here are an extract of the trace :
    2013-05-13 11:01:11,755 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine – created tmp python.cachedir=/tmp/pig_jython_391280432392436532
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/jersey-json-1.8.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/jsp-api-2.1.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/jersey-core-1.8.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/avro-1.5.3.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/log4j-1.2.17.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/guava-11.0.2.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/lib/hadoop/lib/jettison-1.1.jar’

    *sys-package-mgr*: processing new jar, ‘/usr/java/jdk1.7.0_17/jre/lib/ext/sunec.jar’
    *sys-package-mgr*: processing new jar, ‘/usr/java/jdk1.7.0_17/jre/lib/ext/dnsns.jar’
    2013-05-13 11:03:48,291 [main] WARN org.apache.pig.scripting.jython.JythonScriptEngine – pig.cmd.args.remainders is empty. This is not expected unless on testing.
    2013-05-13 11:03:48,664 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine – Register scripting UDF: myfuncs.concat

    Collapse
    #24681

    Larry Liu
    Moderator

    Hi, Vladislav,

    Can you please provide more detail?

    Thanks

    Larry

    Collapse
Viewing 5 replies - 1 through 5 (of 5 total)