Arun Ganesh Jeganathan
Thanks for your reply. I tried this, but it did not work.
The HDP version I was trying was HDP 2.0.2. I had another HDP installation HDP 2.0.6 and the command pig -useHCatalog worked fine there. I noticed that one of the jars /usr/lib/hive/lib/libthrift-0.9.1.jar in HDP 2.0.2 was a older version in HDP 2.0.6 (/usr/lib/hive/lib/libthrift-0.9.0.jar). So I copied all the jars from the /hive/lib library in 2.0.6 to 2.0.2 and the command worked fine.
So I believe this is a jar issue in 2.0.2. Can you have a look or confirm?