Home Forums HDFS WebHDFS error when upload file

Tagged: ,

This topic contains 6 replies, has 4 voices, and was last updated by  tedr 1 year, 3 months ago.

  • Creator
    Topic
  • #24880

    Hi,

    I’m getting the following error message when I upload a File to HDFS using WebHDFS that has spaces on it, (no spaces works great):

    I’m basically using URLEncode for the path of the HTTP request, the Namenode seems to accept this but getting error when make the second Request (PUT) to the Datanode from the initial redirect of the Namenode.

    Am I missing something?

    2013-05-08 09:57:52,989 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs cause:java.net.URISyntaxException: Illegal character in path at index 51: webhdfs://HadoopMaster.demo.vm:50070/usr/UT/File1MB (1).txt
    2013-05-08 09:57:52,989 WARN org.apache.hadoop.hdfs.web.resources.ExceptionHandler: INTERNAL_SERVER_ERROR
    java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1191)
    at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods.put(DatanodeWebHdfsMethods.java:168)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    Caused by: java.security.PrivilegedActionException: java.net.URISyntaxException: Illegal character in path at index 51: webhdfs://HadoopMaster.demo.vm:50070/usr/UT/File1MB (1).txt
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
    … 40 more
    Caused by: java.net.URISyntaxException: Illegal character in path at index 51: webhdfs://HadoopMaster.demo.vm:50070/usr/UT/File1MB (1).txt
    at java.net.URI$Parser.fail(URI.java:2810)
    at java.net.URI$Parser.checkChars(URI.java:2983)
    at java.net.URI$Parser.parseHierarchical(URI.java:3067)
    at java.net.URI$Parser.parse(URI.java:3015)
    at java.net.URI.(URI.java:577)
    at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods$1.run(DatanodeWebHdfsMethods.java:198)
    at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods$1.run(DatanodeWebHdfsMethods.java:168)
    … 43 more

Viewing 6 replies - 1 through 6 (of 6 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #25374

    tedr
    Member

    Hi Ramiro,

    Yes we will log an issue for this, after making sure it’s not already there. Also I will continue to look for a workaround and will get back to you when I find one.

    Thanks,
    Ted.

    Collapse
    #25151

    Hi tedr,

    I have tried escaping spaces with ‘\’, and it looks like the file name now is correct on the HDFS, however I got the file in a sub folder and I still have an Internal Server error response from the HDFS, but there is no error on the NameNode.log.

    for example
    http://Namenode:50070/usr/UT/File1MB 1.txt – > http://Namenode:50070/usr/UT/File1MB\ 1.txt
    and I ended up having the file under a File1MB subfolder with a file name 1.txt

    # hadoop fs -ls /usr/UT/File1MB/
    Found 1 items
    -rw-r–r– 3 hdfs hdfs 1050626 2013-05-10 13:30 /usr/UT/File1MB/ 1.txt.tmp

    However I think the escape shouldn’t be done by the client, but the Hadoop name node, since HTTP web request have standards and RFC when it comes to URI and URL encoding handling .

    So my question for you guys is:
    Are you guys going to create a Issue so it can be fix in future versions? is there is a workaround I can use.

    Let me know and also let me know there is anything else I can help. cheers.

    Collapse
    #25029

    tedr
    Moderator

    Hi Ramiro,

    Have you tried escaping the spaces with ‘\’ ?

    Thanks,
    Ted.

    Collapse
    #25002

    Hi Seth, yes i can read the file using command line, the file seems to be saved correctly, I can even download it using webHDFS, it just the error code from the request that comes back. Not sure what is happening, also not sure if the file integrity is correct among the cluster. I test with a small file.

    Collapse
    #25001

    Seth Lyubich
    Keymaster

    Hi Ramiro,

    We were able to reproduce this. What is interesting is that we can actually see and read file from command line even though WebHDFS exception happened during put. Can you please check to see if you actually can read file from command line after this exception occurred?

    Thanks,
    Seth

    Collapse
    #24881

    Here is the Request/Response sequence:
    **********
    REQUEST:
    PUT http://hadoopmaster.demo.vm:50070/webhdfs/v1/usr/UT/File1MB%20(1).txt?user.name=hdfs&op=CREATE HTTP/1.1
    Host: hadoopmaster.demo.vm:50070
    Content-Length: 0

    RESPONSE:
    HTTP/1.1 307 TEMPORARY_REDIRECT
    Content-Type: application/octet-stream
    Expires: Thu, 01-Jan-1970 00:00:00 GMT
    Set-Cookie: hadoop.auth=”u=hdfs&p=hdfs&t=simple&e=1368061072186&s=S8h6fTDohs5HZ8Uanm/ghaFGavs=”;Path=/
    Location: http://HadoopNode1.demo.vm:50075/webhdfs/v1/usr/UT/File1MB%20(1).txt?op=CREATE&user.name=hdfs&overwrite=false
    Content-Length: 0
    Server: Jetty(6.1.26)
    **********
    REQUEST:
    PUT http://hadoopnode1.demo.vm:50075/webhdfs/v1/usr/UT/File1MB%20(1).txt?op=CREATE&user.name=hdfs&overwrite=false HTTP/1.1
    Host: hadoopnode1.demo.vm:50075
    Content-Length: 1050626
    Expect: 100-continue

    RESPONSE:
    HTTP/1.1 500 Internal Server Error
    Content-Type: application/json
    Transfer-Encoding: chunked
    Server: Jetty(6.1.26)

    A7
    {“RemoteException”:{“exception”:”UndeclaredThrowableException”,”javaClassName”:”java.lang.reflect.UndeclaredThrowableException”,”message”:”Unknown exception in doAs”}}
    0
    **********

    Collapse
Viewing 6 replies - 1 through 6 (of 6 total)