How to use a newer version of parquet jar on AWS EMR - amazon-web-services

Question

The latest EMR 4.1.0 bundles with Hive 1.0.0 and Spark 1.5.0, and Hive 1.0.0 uses parquet-hadoop-bundle-1.5.0.jar while Spark uses parquet-hadoop-1.7.0.jar
Unfortunately the version parquet 1.5.0 can not read files generated by parquet 1.7.0.
I tried to use add jar parquet-hive-bundle-1.7.0.jar in Hive shell but no luck, Hive still used its bundled old Parquet jar.
Then I tried to replace the old jar with the newer jar, however I couldn't find any parquet related jars even using the command sudo find / "*parquet*.jar".
However I copied parquet-hive-bundle-1.7.0.jar to /usr/lib/hive/lib but it didn't work, Hive still used the old parquet jar and couldn't read my parquet files. Normally this way works in Cloudera distribution.
So my question is, where is the parquet jar and how can I replace it with a newer version?

Answer 1

I resolved the issue by adding the new parquet jar location to the environment variable 'HADOOP_CLASSPATH' in environment or .profile or .bashrc file
export HADOOP_CLASSPATH=/var/lib/hive/parquet-hadoop-bundle-1.9.0.jar:$HADOOP_CLASSPATH
Run below command to get the hive command location
which hive
Open 'hive' file under /usr/bin/(Your hive location)
vi /usr/bin/hive
You should see something like below.
Take a backup of the hive file and add an echo command for the HADOOP_CLASSPATH at the end before exec as below.
hive
#!/bin/bash
if [ -d "/usr/hdp/2.5.0.0-1245/atlas/hook/hive" ]; then
if [ -z "${HADOOP_CLASSPATH}" ]; then
export HADOOP_CLASSPATH=/usr/hdp/2.5.0.0-1245/atlas/hook/hive/*
else
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/2.5.0.0-1245/atlas/hook/hive/*
fi
fi
...
if [ -z "${HADOOP_CLASSPATH}" ]; then
export HADOOP_CLASSPATH=${HCATALOG_JAR_PATH}
else
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HCATALOG_JAR_PATH}
fi
echo "Classpath=$HADOOP_CLASSPATH"
exec "${HIVE_HOME}/bin/hive.distro" "$#"
Run hive command to display the classpath.
Once we get the actual classpath, we know where to add the new jar file to take priority over the old parquet jar.
In this case, if we set the HADOOP_CLASSPATH, the classpath of hive will be prefixed with the value in HADOOP_CLASSPATH.
So adding the HADOOP_CLASSPATH value with new parquet jar fixed the issue.
I don't know whether this is a right fix but its working.