I am trying to upgrade my current EMR from 5.30.0 to 6.6.0 or even 5.35.0 to run my python batch script. Whenever I run my python file, even if it is a plain print statement I get the following error on any version of EMR from 5.35 to 6.6.0 . Does anyone have any tips or idea on this issue ?
Traceback (most recent call last):
File "/mnt1/yarn/usercache/hadoop/appcache/application_11111111_0001/container_16111111110_0001_02_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
File "/mnt1/yarn/usercache/hadoop/appcache/application_111111110_0001/container_16111111630_0001_02_000001/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o92.sql.
: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:127)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:237)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:116)
Looks like the issue was in the applications being installed in the EMR. Hadoop and Hive were not installed on EMR and as a result the above error.
Related
I just discovered that my Elastic Beanstalk server has health status with a warning "97% of memory is in use". Because of this I can not deploy updates or ssh and run the django shell. I just receive the following error:
MemoryError
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/sentry_sdk/worker.py", line 70, in start
self._thread.start()
File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/sentry_sdk/integrations/threading.py", line 54, in sentry_start
return old_start(self, *a, **kw) # type: ignore
File "/usr/lib64/python3.7/threading.py", line 852, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
I received more memory errors when trying AWS documented troubleshooting suggestions which required installing a package:
sudo amazon-linux-extras install epel
From here I don't know what else I can do to troubleshoot this issue or how to fix it.
I am using aws-cli version 2.8.8.
Connecting to AWS using LDAP and it is successful.
If I run command aws s3 ls then I get the results.
However when I try run command aws dynamodb list-tables nothing get displayed. Same for aws ec2 describe-instances no response.
When I run same command in debug mode I can see exception in awscli.clidriver file:
Exception details
2022-11-03 12:27:55,762 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "awscli/clidriver.py", line 458, in main
File "awscli/clidriver.py", line 593, in __call__
File "awscli/clidriver.py", line 769, in __call__
My team members uses the same cli version and account and then can access to all data. The issue is with my Mac terminal.
I tried searching for this issue online but no one has reported it. This could be with my terminal but I am not able to identify root cause.
I have a an ASP.NET Core web application which publishes to AWS using ElasticBeanstalk. In order to configure the windows environment I am using .ebextensions, which will copy the vcpp redistributables from S3 and installs them while creating the environment.
When published I am getting the error 'Error occurred during build: Command 01_instlVCx64 failed". Below is the command in my .ebextensions
files:
"c:\\vcpp-redistributables\\vc_redist_x64.exe":
source: https://<bucket_name>.s3.eu-west-2.amazonaws.com/vcpp-redistributables/vc_redist_x64.exe
authentication: S3Access
commands:
01_instlVCx64:
command: c:\\vcpp-redistributables\\vc_redist_x64.exe /q /norestart
Below is the trace back from the logs
2022-03-22 15:31:35,876 [ERROR] Error encountered during build of prebuild_0_GWebApp: Command 01_instlVCx64 failed
Traceback (most recent call last):
File "cfnbootstrap\construction.pyc", line 578, in run_config
File "cfnbootstrap\construction.pyc", line 146, in run_commands
File "cfnbootstrap\command_tool.pyc", line 127, in apply
cfnbootstrap.construction_errors.ToolError: Command 01_instlVCx64 failed
2022-03-22 15:31:35,876 [ERROR] -----------------------BUILD FAILED!------------------------
Could you please let me know what am I missing?
Thanks in advance.
Found the issue couple of days before. So, thought of answering my own question, so that it will be useful for others.
The issue is Elastic beanstalk instance (Windows server 2019) already has VCpp redistributables installed and which is later version that I am trying to install as part of .ebextensions. So, when I tried to install, it failed.
I figured it out, by enabling RDP connection on the EC2 instance that is created as part of Elastic beanstalk and run the scripts manually, which gave a detailed error message.
Hope it helps someone in the future.
Following the instructions in the documentation, I attempt to create my new project, and get the following error:
EDIT: I should not post questions late at night. Added more detail to the terminal output. Prior to this, I verified pip was upgraded, djangocms-installer is installed and the virtualenv was installed.
(djangoenv) [ec2-user#web01 ~]$ djangocms jbi
Creating the project
Please wait while I install dependencies
If I am stuck for a long time, please check for connectivity / PyPi issues
Dependencies installed
Creating the project
The installation has failed.
*****************************************************************
Check documentation at https://djangocms-installer.readthedocs.io
*****************************************************************
Traceback (most recent call last):
File "/home/ec2-user/djangoenv/bin/djangocms", line 8, in <module>
sys.exit(execute())
File "/home/ec2-user/djangoenv/lib/python3.7/site-packages/djangocms_installer/main.py", line 44, in execute
django.setup_database(config_data)
File "/home/ec2-user/djangoenv/lib/python3.7/site-packages/djangocms_installer/django/__init__.py", line 353, in setup_database
output = subprocess.check_output(command, env=env, stderr=subprocess.STDOUT)
File "/usr/lib64/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/usr/lib64/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/home/ec2-user/djangoenv/bin/python', '-W', 'ignore', 'manage.py', 'migrate']' returned non-zero exit status 1.
Using Django inside a virtual environment created on an AWS EC2 instance, running Amazon Linux 2. I'm OK with burning the instance down and using another distro if the issue is the distro.
I've been trying to get the spark-deep-learning library working on my EMR cluster to be able to read images in parallel with Python 2.7. I have been searching for this for quite some time now and I have failed to reach a solution. I have tried setting different configuration settings in the conf for the sparksession and I get the following error when trying to create a SparkSession object
ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
The above was the result when using jupyter notebook.
I tried submitted the py file with spark submit and adding the jar I need to use as a value for --jars, --driver-class-path and for the --conf spark.executor.extraClassPath as discussed by this link.Here is the code I submit along with the resulting import error :
bin/spark-submit --jars /home/hadoop/spark-deep-learning-0.2.0-spark2.1-s_2.11.jar /
--driver-class-path /home/hadoop/spark-deep-learning-0.2.0-spark2.1-s_2.11.jar /
--conf spark.executor.extraClassPath=/home/hadoop/spark-deep-learning-0.2.0-spark2.1-s_2.11.jar /
/home/hadoop/RunningCode6.py
Traceback (most recent call last):
File "/home/hadoop/RunningCode6.py", line 74, in <module>
from sparkdl import KerasImageFileTransformer
ImportError: No module named sparkdl
The library works fine in a standalone mode, but I keep getting either one of the above stated errors when I use the cluster mode.
I really hope someone can help me solve this because I've been staring at it for weeks now and I need to get it working
Thanks!