How to have some Fabric tasks only run once locally while others run on all hosts - fabric

In my fabric scripts I have the following problem. I have a main task called autodeploy. Within this task I have some tasks that I only want to run once, locally. all remote tasks should run on each of the hosts of the host list.
env.roledefs ={
'testing': ['t-server-01', 't-server-02']
'staging': ['s-server-01', 's-server-02']
'live': ['l-server-01', 'l-server-02']
}
def localtask1():
# download artifact
def localtask2():
# cleanup locally
def remotetask():
# deploy artifact to all hosts
def autodeploy():
localtask1() # run this task only once, locally
remotetask() # run this task on all hosts
localtask2() # run this task only once
The call is the following. I want to pass the role as an attribute.
fab -R test autodeploy

Use the execute function inside the wrapper function autodeploy, and specify a host list for the remote task.
For the other two you can call them with execute, like for the remote task, or directly. Use the local function inside them and you'll be fine, and not need to have ssh on localhost.
Docs are here for how best ot use the new execute function.
EDIT
Since you mention a different use case in the comments I'll mock up how you'd do that, from bits in the documentation given already, adding the param passing section
code:
#copy above
#redefine this one
def autodeploy(role_from_arg):
localtask1()
execute(remotetask, role=role_from_arg)
localtask2()
#calls like fab autodeploy:testing

Use the runs_once decorator.
#runs_once
def localtask1():
local('command')

You can abuse the hosts decorator to force a single task to run once only, by specifying "localhost" as the host.
Example:
#fabric.decorators.hosts("localhost")
def localtask1():
# download artefact

Related

Input and Ouput to ECS task in Step function

Have previously worked with lambda orchestration using AWS step function. This has been working very well. Setting the result_path of each lambda will pass along arguments to subsequent lambda.
However, I now need to run a fargate task and then pass along arguments from that fargate task to subsequent lambdas. I have created a python script that acts as the entrypoint in the container definition. Obviously in a lambda function the handler(event, context) acts as the entrypoint and by defining a return {"return_object": "hello_world"} its easy to pass a long a argument to the next state of the state machine.
In my case though, I have task definition with a container definition created from this Dockerfile:
FROM python:3.7-slim
COPY my_script.py /my_script.py
RUN ln -s /python/my_script.py /usr/bin/my_script && \
chmod +x /python/my_script.py
ENTRYPOINT ["my_script"]
Hence, I am able to invoke the state machine and it will execute my_script as intended. But how do I get the output from this python script and pass it along to another state in the state machine?
I have found some documentation on how to pass along inputs, but no example of passing along outputs.
To get output from an ECS/Fargate task, I think you have to use the Task Token Integration instead of Run Job (Sync) which is usually recommended for Fargate tasks. You can pass the token as a container override ("TASK_TOKEN": "$$.Task.Token"). Then inside your image you need some logic like this:
client = boto3.client('stepfunctions')
client.send_task_success(
taskToken=os.environ["TASK_TOKEN"],
output=output
)
to pass it back.

Does it make sense to run a non web application on cloud run?

I see that all of the examples per the documentation use some form of a simple web application (For example, Flask in Python). Is it possible to use cloud run as a non web application? For example, deploy cloud run to use a python script and then use GCP Scheduler to invoke cloud run every hour to run that script? Basically my thinking for this is to avoid having to deploy and pay for Compute Engine, and only pay for when the cloud run container is invoked via the scheduler.
It's mandatory to answer to HTTP request. It's the contract of Cloud Run
Stateless (no volume attached to the container)
Answer to HTTP request
However, if you already have a python script, it's easy to wrap it in a flask webserver. Let's say, you have something like this (I assume that the file name is main.py -> important for the Dockerfile at the end)
import ....
var = todo(...)
connect = connect(...)
connect(var)
Firstly, wrap it in a function like this
import ....
def my_function(request):
var = todo(...)
connect = connect(...)
connect(var)
return 'ok',200
Secondly, add a flask server
from flask import Flask, request
import os
import ....
app = Flask(__name__)
#app.route('/')
def my_function(request):
var = todo(...)
connect = connect(...)
connect(var)
return 'ok',200
if __name__ == "__main__":
app.run(host='0.0.0.0',port=int(os.environ.get('PORT',8080)))
Add flask in your requirements.txt
Build a standard container, here an example of Dockerfile
FROM python:3-alpine
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT 8080
CMD [ "python", "main.py" ]
Build (with Cloud Build for example) and deploy the service on Cloud Run
Now you have an URL, that you can call with Cloud Scheduler.
Be careful, the max request duration is, for now, limited to 15 minutes (soon 4x more) and limited to 2vCPU and 2Gb of memory (again, soon more).
It depends what is being installed in the container image, as there is no requirement that one would have to install a web-server. For example, with such an image I can build Android applications, triggered whenever a repository changes (file excludes recommend) ...and likely could even run a head-less Android emulator for Gradle test tasks and publish test results to Pub/Sub (at least while the test-suite wouldn't run for too long). I mean, one has to understand the possibilities of Cloud Build to understand what Cloud Run can do.
I've struggled with deployment my function which no need to handle any request in Cloud Run by putting functions in Flask app and found out that Cloud Run provides us 2 kinds of jobs, services and jobs.
Illustration from codelabs
From cloud run jobs documentation,
This page describes how to create and update Cloud Run jobs from an existing container image. Unlike services, which listen for requests, a job does not serve requests but only runs its tasks and exits when finished.
After you create or update a job, you can execute the job as a one-off, on a schedule or as part of a workflow. You can manage individual job executions and view the execution logs.
You may see that there are two tabs in Cloud Run console. I am not sure when Cloud Run jobs started.
See cloud run console

Cannot assign instance name to concurrent workflow in Informatica

In Informatica, I can start a workflow but cannot get it to recognize my instance name in the session log and Workflow Monitor.
The workflow starts but in the session log it displays this:
Workflow wf_Tenter image description hereemp started with run id [22350], run instance name [], run type [Concurrent Run with Un[enter image description here][1]ique Instance Name]
Instance name is blank.
My command is:
pmcmd startworkflow -sv <service> -d <domain> -u <user> -p <password> -f <folder> -rin INST1 -paramfile <full param file path name> wf_Temp
I have edited the workflow and selected the checkbox Configure Current Execution. Inside Configure Concurrent Execution button, I have created three instances: INST1, INST2, INST3, but without any associated parameter files. All parameter files are blank.
I understand, I think, that in order to start a workflow with PMCMD I must pass in one of the configured instance names (i.e. INST1, INST2, INST3, etc.)
If I execute the PMCMD command from Putty a second time to see the second instance run, I receive a message that workflow is still running and I have to wait? Why? I have checked the Concurrent Workflow box in the workflow.
ERROR: Workflow [wf_Temp]: Could not start execution of this workflow because the current run on this Integration Service has not completed yet.
Disconnecting from Integration Service
So, I think I'm close, but am missing something. The workflow runs with the parameter file I pass in PMCMD but the instance name seems to be ignored.
Further. Do, I have to pre-configure instance names in the Workflow manager? Is the PMCMD instance and parameter file parameters enough? It doesn't seem quite so dynamic if Instances have to be pre-defined in the workflows.
Thanks.
#MacieJG
Here's the screenshots from Putty when I run the command. You can see the instance name DALLAS is being passed through the PMCMD OK. No combination ever gets the Instance name. I did not include the pics of your suggested Test 1, but results were same.. still no instance.
Here's my complete test as requested in a comment above. I tried my best to put everything you may need here, but if I missed anything, just let me know. So here goes...
I've created a very simple workflow to run with instance name. It uses a timer to wait and a command tast to write the instance name to a file:
The concurrent execution has been set up in the most simple way:
Now, I've prepared the followig batch to run the workflow (just user & password removed):
SET "PMCMD=C:\Informatica\9.5.1\clients\PowerCenterClient\CommandLineUtilities\PC\server\bin\pmcmd"
%PMCMD% startworkflow -sv Dev_IS -d Domain_vic-vpc -u ####### -p ####### -f Dev01 -rin GLASGOW wf_Instance_Test
%PMCMD% startworkflow -sv Dev_IS -d Domain_vic-vpc -u ####### -p ####### -f Dev01 -rin FRANKFURT wf_Instance_Test
%PMCMD% startworkflow -sv Dev_IS -d Domain_vic-vpc -u ####### -p ####### -f Dev01 -rin GLASGOW wf_Instance_Test
It runs three instances, two of them with same name, just to test it. I run the batch the following way to capture the output:
pmStartTestWF.bat > c:\MG\pmStartTestWF.log
Once I execute it, here what I see in workflow monitor:
Just as expected, three instances executed and properly displayed. File output looks fine as well:
The output of pmcmd can be found here. Full definition of my test workflow is available here.
I really hope this will help you somehow. Feel free to let me know if you'd find anything missing here. Good luck!
You don't need to pre-configure instance names in workflow. Passing the instance name in pmcmd along with parameter filename is enough.
try this: pmcmd startworkflow -sv (service) -d (domain) -u (user) -p (password) -f (folder) -paramfile (full param file path name) -rin INST1 wf_Temp
To be precise: when you configure Concurrent Execution, you can specify if you:
allow concurrent run with same instance name
allow concurrent run only with unique instance name
In addition to that you may, but don't have to, indicate which instance should use which parameter file, so it won't be need to mention it while executing. But that's a separate feature.
Now, if you've chosen the first one, you will be able to invoke the WF multiple times with the very same command. If you've chosen the second one and try this, you will get the 'WF is already running' error.
The trouble is that your example seems correct at first glance. As per the log message:
Workflow wf_Temp started with run id [22350], run instance name [], run type [Concurrent Run with Unique Instance Name]
So you're allowing unique instances only. It seems that the instance name has not been used. First execution does not set the instance name, so similar second execution won't use it either and will get rejected as this is the same instance name (i.e. None).
You may try to change the setting to Allow concurrent run with same instance name, this shall allow the secon execution, but does not solve the main issue. For some reason the instance name does not get passed.
Please verify your command against the docs referenced below. Try to match the order perhaps. Please share some more info if it still fails.
Looking at the docs:
pmcmd StartWorkflow
<<-service|-sv> service [<-domain|-d> domain] [<-timeout|-t> timeout]>
<<-user|-u> username|<-uservar|-uv> userEnvVar>
<<-password|-p> password|<-passwordvar|-pv> passwordEnvVar>
[<<-usersecuritydomain|-usd> usersecuritydomain|<-usersecuritydomainvar|-usdv>
userSecuritydomainEnvVar>]
[<-folder|-f> folder]
[<-startfrom> taskInstancePath]
[<-recovery|-norecovery>]
[<-paramfile> paramfile]
[<-localparamfile|-lpf> localparamfile]
[<-osprofile|-o> OSUser]
[-wait|-nowait]
[<-runinsname|-rin> runInsName]
workflow

manage.py: cannot connect to X server

I have used PyQt4.QtWebkit to crawl the web page in my django application.In the production environment that module doesn't work to crawl it.it throws the error "manage.py: cannot connect to X server"
My Qt class :
class Render(QWebPage):
def __init__(self, url):
self.app = QApplication(sys.argv)
QWebPage.__init__(self)
self.loadFinished.connect(self._loadFinished)
self.mainFrame().load(QUrl(url))
self.app.exec_()
def _loadFinished(self, result):
self.frame = self.mainFrame()
self.app.quit()
calling from django-shell:
r = Render(url)
when i call this "Render" class through django with the Django-shell(python manage.py shell) the render function throws the error.
could you please help me on this?
The Reason is "Xvfb"
i need to run my python program in bash shell with xvfb(X virtual frame buffer)
likewise,
ubuntu#localhost$ xvfb-run python webpage_scrapper.py http://www.google.ca/search?q=navaspot
It gives the result.
Now My requirement is i need to execute this shell command in python and waiting for tine to collect the result.I have to process the result.
Could you please suggest me for executing this command on python effectively.
Seems like environment variables for X display are not set and that's the reason you get such error. It can occur because you're running script from environment, that isn't bound to X display (ssh to server).
Try adding display variable:
DISPLAY=:0.0 python manage.py script
It is also possible to set DISPLAY environment variable from python. You may set it before calling the PyQt4:
import os
os.putenv('DISPLAY', ':0.0')
It's also may not be possible to run PyQt4.QtWebkit if your production environment doesn't have X server running.
Generally on headless machines, the DISPLAY variable is absent or misconfigured. To work on such machines, you can use the following approach. As a example for Ubuntu 14.04-LTS machines:
First install X server:
sudo apt-get install xserver-xorg
Now start the X server (say at :0):
sudo /usr/bin/X :0&
You can use process managers like supervisor to handle the above process.
Now just set the DISPLAY environment variable and make sure it is available to any processes you are running which depend on this,
DISPLAY=:0 python manage.py
The way you provide the environment variables to your application is upto you.

How to run a deploy command on remote host from PyCharm?

I am looking for a way to simplify remote deployment of a django application directly from PyCharm.
Even if deploying the files itself works just file with the remote host and upload, I was not able to find a way to run the additional commands on the server site (like manage.py syncdb).
I am looking for a fully automated solution, one that would work at single click (or command).
I don't know much about PyCharm so maybe you could do something from the IDE, but I think you'll probably want to take a look at the fabric project (http://docs.fabfile.org/en/1.0.1/index.html)
It's a python deployment automation tool that's pretty great.
Here is one of my fabric script files. Note that I make a lot of assumptions (This is my own that I use) that completely depend on how you want to set up your project, such as I use virtualenv, pip, and south as well as my own personal preference for how to deploy and where to deploy to.
You'll likely want to rework or simplify it to meet your needs.
You may use File > Settings > Tools > External Tools to run arbitrary external executable files. You may write a small command that connects over SSH and issues a [set of] command. Then the configured tool would be executable
For example, in my project based on tornado, I run the instances using supervisord, which, according to answer here, cannot restart upon code change.
I ended up writing a small tool on paramiko, that connects via ssh and runs supervisorctl restart. The code is below:
import paramiko
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-s",
action="store",
dest="server",
help="server where to execute the command")
parser.add_option("-u",
action="store",
dest="username")
parser.add_option("-p",
action="store",
dest="password")
(options, args) = parser.parse_args()
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect(hostname=options.server, port=22, username=options.username, password=options.password)
command = "supervisorctl reload"
(stdin, stdout, stderr) = client.exec_command(command)
for line in stdout.readlines():
print line
client.close()
External Tool configuration in Pycharm:
program: <PYTHON_INTERPRETER>
parameters: <PATH_TO_SCRIPT> -s <SERVERNAME> -u <USERNAME> -p <PASSWORD>