Is there any way to fetch the application ID when running - for example - the wordcount example with the yarn command?
I wish to initiate a job from another process with the yarn command, and monitor the status of the job through the YARN REST API.
(Using Hadoop 2.4.0)
You can use yarn application -list command to get the list of all the applications.
Specifically if you want to get the list of all the applications, which are currently in RUNNING state, you can execute the following command:
yarn application -list -appStates RUNNING
If you already know the application ID, then you can query the status of the application using the following command:
yarn application -status <application ID>
For e.g.
yarn application -status application_1448359154956_0001
I get following application report (in this case application was KILLED by user):
Application Report :
Application-Id : application_1448359154956_0001
Application-Name : distcp
Application-Type : MAPREDUCE
User : mballur
Queue : default
Start-Time : 1448359237581
Finish-Time : 1448359419592
Progress : 100%
State : KILLED
Final-State : KILLED
Tracking-URL : http://mballur.fareast.corp.microsoft.com:8088/cluster/ap
p/application_1448359154956_0001
RPC Port : -1
AM Host : N/A
Aggregate Resource Allocation : 1652876 MB-seconds, 1337 vcore-seconds
Log Aggregation Status : NOT_START
Diagnostics : Application killed by user.
You can parse this output to get State and Progress of the application.
Checkout http://hadoop.apache.org/docs/r2.4.0/api/org/apache/hadoop/yarn/client/api/YarnClient.html. Should have a something to get the application ID you need.
Related
I am working on a scenario where I need to stop my spring boot application whenever aws stack is updated.
I used command
"commands": {
"1 - shutdown app " : {
"command" : {"Fn::Join": ["",["systemctl stop ",{"Ref":"appname"}]]}}
This command is working fine if already have stack available. However, In case of creating stack from scratch. The command is not working because application is not available yet.
I am looking for a condition where if we updating the stack above command will run or when we creating the stack, command will not run.
one more alternative come into my mind to check for running java process using ps -ef | grep java .
Is there any way to check the output of grep in cloudformation. If PID is available, I will stop the app otherwise, will not run command.
Let me know if you have any other approach to find solution for me.
I have many EC2 instances that retain Celery jobs for processing. To efficiently start the overall task of completing the queue, I have tested AWS-RunBashScript in AWS' SSM with a BASH script that calls a Python script. For example, for a single instance this begins with sh start_celery.sh.
When I run the command in SSM, this is the following output (compare to other output below, after reading on):
/home/ec2-user/dh2o-py/venv/local/lib/python2.7/dist-packages/celery/utils/imports.py:167:
UserWarning: Cannot load celery.commands extension u'flower.command:FlowerCommand':
ImportError('No module named compat',)
namespace, class_name, exc))
/home/ec2-user/dh2o-py/tasks/task_harness.py:49: YAMLLoadWarning: calling yaml.load() without
Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
task_configs = yaml.load(conf)
Running a worker with superuser privileges when the worker accepts messages serialized with pickle is a very bad idea!
If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).
User information: uid=0 euid=0 gid=0 egid=0
failed to run commands: exit status 1
Note that only warnings are thrown. When I SSH to the same instance and run the same command (i.e. sh start_celery.sh), the following (same) output results BUT the process runs:
I have verified that the process does NOT run when doing this via SSM, and I have no idea why. As a work-around, I tried running the sh start_celery.sh command with bootstrapping in user data for each EC2, but that failed too.
So, why does SSM fail to actually run the process that I succeed in doing by actually via SSH to each instance running identical commands? The details below relate to machine and Python configuration:
I have started a peer and membersrvc container with docker compose. They have started successfully. I deploying example02 chaincode from CLI (tried REST also). I get a success message. When i try to query the chaincode, i am getting Error when querying chaincode: Error:Failed to launch chaincode spec(Could not get deployment transaction for mycc - LedgerError - ResourceNotFound: ledger: resource not found)"
If you are trying to deploy the chaincode in dev mode, you first need to register the chaincode.
(Registration is only required in dev mode and not for production mode)
To register your chaincode on windows 10 machine in docker container :
open command prompt and go to bash shell using docker command
docker exec -it [peer container id] /bin/bash
Browse to chainocde directory and register it using
CORE_CHAINCODE_ID_NAME=mycc CORE_PEER_ADDRESS=127.0.0.1:7051 ./chaincode_example02
Now you would see register successful message : “Received REGISTERED, ready for invocations” and is ready to deploy, invoke and query in dev mode
Note: Leave the window as with register handler open, closing it would deregister the chaincode.
Waiting for a few minutes after the chaincode deployment might produce different results when querying. As described here, it could take a couple of minutes for chaincode to deploy. Another suggestion mentioned is to review the chaincode container log to determine if there are problems communicating with a peer.
It is also possible that the chaincode deployment was not successful. The log for the peer where the chaincode deployment was initiated could be reviewed to determine if this provides any insight.
There are also a couple of prior posts that are similar and might help.
How to debug chaincode? LedgerError - ResourceNotFound
Hyperledger : Deploying chaincode successful. But, cannot query - says ResourceNotFound
Let's say I want to check the yarn logs with the command "yarn logs" but I can't access to the ApplicationID of a MapReduce job neither through the output or the spark context of the code. How can I check the last Application ID's that have been executed?
To get the list of all the applications submitted so far, you can use the following command:
yarn application -list -appStates ALL
You can also filter the applications, based on the state (possible states: NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILEDand KILLED).
For e.g. to get the list of all the "FAILED" applications, you can execute the following command:
yarn application -list -appStates FAILED
As we all know by now, the only way to run tests on iOS is by using the simulator. My problem is that we are running jenkins and the iOS builds are running on a slave (via SSH), as a result running xcodebuild can't start the simulator (as it runs headless). I've read somewhere that it should be possible to get this to work with SimLauncher (gem sim_launcher). But I can't find any info on how to set this up with xcodebuild. Any pointers are welcome.
Headless and xcodebuild do not mix well. Please consider this alternative:
You can configure the slave node to launch via jnlp (webstart). I use a bash script with the .command extension as a login item (System Preferences -> Users -> Login Items) with the following contents:
#!/bin/bash
slave_url="https://gardner.company.com/jenkins/jnlpJars/slave.jar"
max_attempts=40 # ten minutes
echo "Waiting to try again. curl returneed $rc"
curl -fO "${slave_url}" >>slave.log
rc=$?
if [ $rc -ne 0 -a $max_attempts -gt 0 ]; then
echo "Waiting to try again. curl returneed $rc"
sleep 5
curl -fO "${slave_url}" >>slave.log
rc=$?
if [ $rc -eq 0 ]; then
zip -T slave.jar
rc=$?
fi
let max_attempts-=1
fi
# Simulator
java -jar slave.jar -jnlpUrl https://gardner.company.com/jenkins/computer/buildmachine/slave-agent.jnlp -secret YOUR_SECRET_KEY
The build user is set to automatically login. You can see the arguments to the slave.jar app by executing:
gardner:~ buildmachine$ java -jar slave.jar --help
"--help" is not a valid option
java -jar slave.jar [options...]
-auth user:pass : If your Jenkins is security-enabled, specify
a valid user name and password.
-connectTo HOST:PORT : make a TCP connection to the given host and
port, then start communication.
-cp (-classpath) PATH : add the given classpath elements to the
system classloader.
-jar-cache DIR : Cache directory that stores jar files sent
from the master
-jnlpCredentials USER:PASSWORD : HTTP BASIC AUTH header to pass in for making
HTTP requests.
-jnlpUrl URL : instead of talking to the master via
stdin/stdout, emulate a JNLP client by
making a TCP connection to the master.
Connection parameters are obtained by
parsing the JNLP file.
-noReconnect : Doesn't try to reconnect when a communication
fail, and exit instead
-proxyCredentials USER:PASSWORD : HTTP BASIC AUTH header to pass in for making
HTTP authenticated proxy requests.
-secret HEX_SECRET : Slave connection secret to use instead of
-jnlpCredentials.
-slaveLog FILE : create local slave error log
-tcp FILE : instead of talking to the master via
stdin/stdout, listens to a random local
port, write that port number to the given
file, then wait for the master to connect to
that port.
-text : encode communication with the master with
base64. Useful for running slave over 8-bit
unsafe protocol like telnet
gardner:~ buildmachine$
For a discussion about OSX slaves and how the master is launched please see this Jenkins bug: https://issues.jenkins-ci.org/browse/JENKINS-21237
Erik - I ended up doing the items documented here:
Essentially:
The first problem, is that you do have to have the user that runs the builds also logged in to the console on that Mac build machine. It needs to be able to pop up the simulator, and will fail if you don’t have a user logged in — as it can’t do this entirely headless without a display.
Secondly, the XCode Developer tools requires elevated privileges in order to execute all of the tasks on the Unit tests. Sometimes you may miss seeing it, but without these, the Simulator will give you an authentication prompt that never clears.
A first solution to this (on Mavericks) is to run:
sudo security authorizationdb write system.privilege.taskport allow
This will eliminate one class of these authentication popups. You’ll also need to run:
sudo DevToolsSecurity --enable
Per Apple’s man page on this tool:
On normal user systems, the first time in a given login session that
any such Apple-code-signed debugger or performance analysis tools are
used to examine one of the user’s processes, the user is queried for
an administator password for authorization. DevToolsSecurity tool to
change the authorization policies, such that a user who is a member of
either the admin group or the _developer group does not need to enter
an additional password to use the Apple-code-signed debugger or
performance analysis tools.
Only issue is that these same things seem to be broken once I upgraded to Xcode 6. Back to the drawing board....