I want to setup a cron job to delete screenshots which are older than two minutes and the cron job to run every two minutes.
Contents of schedule.rb file
every 2.minutes do
runner 'screenshot.delete_old', output: {error: "#{path}/log/error.log", standard: "#{path}/log/cron.log"}
end
Contents of Model Screenshot.rb file
def self.delete_old
# #old_screenshots = Screenshot.where('Screenshot.created_at < ?', Time.now)
##old_screenshots.delete_all
##screenshot.delete_old
#Screenshot.delete
screenshot_ids = screenshot.find(:all, :conditions => ["created_at ", 2.minutes.ago])
if screenshot_ids.size > 0
screenshot.destroy(screenshot_ids)
puts "#{screenshot_ids.size} screenshots have been deleted!"
end
end
I updated the crontab file and ran the rails server expecting the screeshots to be deleted. But it is not working. Could someone help me with this? Thanks.
Classes in Ruby should have capital first letter, maybe you should try:
every 2.minutes do
runner 'Screenshot.delete_old', output: {error: "#{path}/log/error.log", standard: "#{path}/log/cron.log"}
end
and your model:
def self.delete_old
screenshot_ids = Screenshot.where('created_at < ?', 2.minutes.ago))
if screenshot_ids.size > 0
Screenshot.destroy(screenshot_ids)
puts "#{screenshot_ids.size} screenshots have been deleted!"
end
end
Related
We have a unit test suite, written in RSpec. We have some failed tests that is a lot actually.
What I am looking for is a script or a magic command, to mark all the failed tests as skipped, so I don't have to go over them one by one and mark them as skipped.
I found this awesome script that do exactly what I need:
https://gist.github.com/mcoms/77954d191bde31d4677872d2ab3d0cd5
Copying the contents here, in case the original gist is deleted:
# frozen_string_literal: true
class CustomFormatter
RSpec::Core::Formatters.register self, :example_failed
def initialize(output)
#output = output
end
def example_failed(notification)
tf = Tempfile.new
File.open(notification.example.metadata[:file_path]) do |f|
counter = 1
while (line = f.gets)
if counter == notification.example.metadata[:line_number]
line.sub!('it', 'skip')
line.sub!('scenario', 'skip')
#output << line
end
tf.write line
counter += 1
end
end
tf.close
FileUtils.mv tf.path, notification.example.metadata[:file_path]
end
end
Should be relatively straightforward. RSpec lists failing specs like this
rspec ./spec/models/user.rb:67 # User does this thing
rspec ./spec/models/post.rb:13 # Post does another thing
rspec ./spec/models/rating.rb:123 # Rating does something else entirely
File name and line number point to the opening line of the test, the one with it ... do.
Write a script that
extracts file names and line numbers from the failure output
opens those files, goes to the specified line
and replaces it with xit.
I am new to GitPython and I am trying to get the content of a file within a commit. I am able to get each file from a specific commit, but I am getting an error each time I run the command. Now, I know that the file exist in GitPython, but each time I run my program, I am getting the following error:
returned non-zero exit status 1
I am using Python 2.7.6 and Ubuntu Linux 14.04.
I know that the file exist, since I also go directly into Git from the command line, check out the respective commit, search for the file, and find it. I also run the cat command on it, and the file contents are displayed. Many times when the error shows up, it says that the file in question does not exist. I am trying to go through each commit with GitPython, get every blob or file from each individual commit, and run an external Java program on the content of that file. The Java program is designed to return a string to Python. To capture the string returned from my Java code, I am also using subprocess.check_output. Any help will be greatly appreciated.
I tried passing in the command as a list:
cmd = ['java', '-classpath', '/home/rahkeemg/workspace/CSCI499_Java/bin/:/usr/local/lib/*:', 'java_gram.mainJava','absolute/path/to/file']
subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=False)
And I have also tried passing the command as a string:
subprocess.check_output('java -classpath /home/rahkeemg/workspace/CSCI499_Java/bin/:/usr/local/lib/*: java_gram.mainJava {file}'.format(file=entry.abspath.strip()), shell=True)
Is it possible to access the contents of a file from GitPython?
For example, say there is a commit and it has one file foo.java
In that file is the following lines of code:
foo.java
import java.io.FileInputStream;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
public class foo{
public static void main(String[] args) throws Exception{}
}
I want to access everything in the file and run an external program on it.
Any help would be greatly appreciated. Below is a piece of the code I am using to do so
#! usr/bin/env python
__author__ = 'rahkeemg'
from git import *
import git, json, subprocess, re
git_dir = '/home/rahkeemg/Documents/GitRepositories/WhereHows'
# make an instance of the repository from specified path
repo = Repo(path=git_dir)
heads = repo.heads # obtain the different repositories
master = heads.master # get the master repository
print master
# get all of the commits on the master branch
commits = list(repo.iter_commits(master))
cmd = ['java', '-classpath', '/home/rahkeemg/workspace/CSCI499_Java/bin/:/usr/local/lib/*:', 'java_gram.mainJava']
# start at the very 1st commit, or start at commit 0
for i in range(len(commits) - 1, 0, -1):
commit = commits[i]
commit_num = len(commits) - 1 - i
print commit_num, ": ", commit.hexsha, '\n', commit.message, '\n'
for entry in commit.tree.traverse():
if re.search(r'\.java', entry.path):
current_file = str(entry.abspath.strip())
# add the current file or blob to the list for the command to run
cmd.append(current_file)
print entry.abspath
try:
# This is the scenario where I pass arguments into command as a string
print subprocess.check_output('java -classpath /home/rahkeemg/workspace/CSCI499_Java/bin/:/usr/local/lib/*: java_gram.mainJava {file}'.format(file=entry.abspath.strip()), shell=True)
# scenario where I pass arguments into command as a list
j_response = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=False)
except subprocess.CalledProcessError as e:
print "Error on file: ", current_file
# Use pop on list to remove the last string, which is the selected file at the moment, to make place for the next file.
cmd.pop()
First of all, when you traverse the commit history like this, the file will not be checked out. All you get is the filename, maybe leading to the file or maybe not, but certainly it will not lead to the file from different revision than currently checked-out.
However, there is a solution to this. Remember that in principle, anything you could do with some git command, you can do with GitPython.
To get file contents from specific revision, you can do the following, which I've taken from that page:
git show <treeish>:<file>
therefore, in GitPython:
file_contents = repo.git.show('{}:{}'.format(commit.hexsha, entry.path))
However, that still wouldn't make the file appear on disk. If you need some real path for the file, you can use tempfile:
f = tempfile.NamedTemporaryFile(delete=False)
f.write(file_contents)
f.close()
# at this point file with name f.name contains contents of
# the file from path entry.path at revision commit.hexsha
# your program launch goes here, use f.name as filename to be read
os.unlink(f.name) # delete the temp file
Trying to get Carrierwave multiple file uploads working. I'm following the documentation on the homepage. When I try to upload a file or multiple, I get an no implicit conversion of nil to string
That error is coming from this method in the Carrierwave gem found in uploaders/cache.rb
def workfile_path(for_file=original_filename)
File.join(CarrierWave.tmp_path, #cache_id, version_name.to_s, for_file)
end
The issue is that the original_file is nil. I've tried to trace the issue but can't find where the issue is really beginning. One thing that is odd is that I am following some source code from this repo
https://github.com/bobintornado/sample-gallery-app-with-carrierwave
The sample app is working and you can do multiple uploads. The difference though is that when cache! method is called the new_file is an Array where in sample app that's working it's Http::UploadedFile
Here's the cache method
def cache!(new_file = sanitized_file)
new_file = CarrierWave::SanitizedFile.new(new_file)
return if new_file.empty?
raise CarrierWave::FormNotMultipart if new_file.is_path? && ensure_multipart_form
self.cache_id = CarrierWave.generate_cache_id unless cache_id
#filename = new_file.filename
self.original_filename = new_file.filename
begin
# first, create a workfile on which we perform processings
if move_to_cache
#file = new_file.move_to(File.expand_path(workfile_path, root), permissions, directory_permissions)
else
#file = new_file.copy_to(File.expand_path(workfile_path, root), permissions, directory_permissions)
end
with_callbacks(:cache, #file) do
#file = cache_storage.cache!(#file)
end
ensure
FileUtils.rm_rf(workfile_path(''))
end
end
Here are my initial params
"coach"=>{"name"=>"ben", "title"=>"ceo", "description"=>"head dude",
"photos"=>[
#<ActionDispatch::Http::UploadedFile:0x007fc9a5235c78 #tempfile=#<Tempfile:/var/folders/sb/t6rry5j928l3sy96nkhy9f840000gn/T/RackMultipart20160113-67635-avg8ef.jpg>, #original_filename="benn-1.jpg", #content_type="image/jpeg", #headers="Content-Disposition: form-data; name=\"coach[photos][]\"; filename=\"benn-1.jpg\"\r\nContent-Type: image/jpeg\r\n">,
#<ActionDispatch::Http::UploadedFile:0x007fc9a5235c50 #tempfile=#<Tempfile:/var/folders/sb/t6rry5j928l3sy96nkhy9f840000gn/T/RackMultipart20160113-67635-r8bdxp.jpg>, #original_filename="benn-2.jpg", #content_type="image/jpeg", #headers="Content-Disposition: form-data; name=\"coach[photos][]\"; filename=\"benn-2.jpg\"\r\nContent-Type: image/jpeg\r\n">
]}
Sorry this isn't terribly helpful, but I burned my current working branch and started from the beginning and now everything is working. Not sure what I did differently. If you're running into the same issue I recommend starting over and following this tutorial:
Multiple-Images-Uploading-With-CarrierWave-and-PostgreSQL-Array
I am using Python 2.7 and Jenkins.
I am writing some code in Python that will perform a checkin and wait/poll for Jenkins job to be complete. I would like some thoughts on around how I achieve it.
Python function to create a check-in in Perforce-> This can be easily done as P4 has CLI
Python code to detect when a build got triggered -> I have the changelist and the job number. How do I poll the Jenkins API for the build log to check if it has the appropriate changelists? The output of this step is a build url which is carrying out the job
How do I wait till the Jenkins job is complete?
Can I use snippets from the Jenkins Rest API or from Python Jenkins module?
If you need to know if the job is finished, the buildNumber and buildTimestamp are not enough.
This is the gist of how I find out if a job is complete, I have it in ruby but not python so perhaps someone could update this into real code.
lastBuild = get jenkins/job/myJob/lastBuild/buildNumber
get jenkins/job/myJob/lastBuild/build?token=gogogo
currentBuild = get jenkins/job/myJob/lastBuild/buildNumber
while currentBuild == lastBuild
sleep 1
thisBuild = get jenkins/job/myJob/lastBuild/buildNumber
buildInfo = get jenkins/job/myJob/[thisBuild]/api/xml?depth=0
while buildInfo["freeStyleBuild/building"] == true
buildInfo = get jenkins/job/myJob/[thisBuild]/api/xml?depth=0
sleep 1
ie. I found I needed to A) wait until the build starts (new build number) and B) wait until the building finishes (building is false).
You can query the last build timestamp to determine if the build finished. Compare it to what it was just before you triggered the build, and see when it changes. To get the timestamp, add /lastBuild/buildTimestamp to your job URL
As a matter of fact, in your Jenkins, add /lastBuild/api/ to any Job, and you will see a lot of API information. It even has Python API, but I not familiar with that so can't help you further
However, if you were using XML, you can add lastBuild/api/xml?depth=0 and inside the XML, you can see the <changeSet> object with list of revisions/commit messages that triggered the build
Simple solution using invoke and block_until_complete methods (tested with Python 3.7)
import jenkinsapi
from jenkinsapi.jenkins import Jenkins
...
server = Jenkins(jenkinsUrl, username=jenkinsUser,
password=jenkinsToken, ssl_verify=sslVerifyFlag)
job = server.create_job(jobName, None)
queue = job.invoke()
queue.block_until_complete()
Inpsired by a test method in pycontribs
This snippet starts build job and wait until job is done.
It is easy to start the job but we need some kind of logic to know when job is done. First we need to wait for job ID to be applied and than we can query job for details:
from jenkinsapi import jenkins
server = jenkins.Jenkins(jenkinsurl, username=username, password='******')
job = server.get_job(j_name)
prev_id = job.get_last_buildnumber()
server.build_job(j_name)
while True:
print('Waiting for build to start...')
if prev_id != job.get_last_buildnumber():
break
time.sleep(3)
print('Running...')
last_build = job.get_last_build()
while last_build.is_running():
time.sleep(1)
print(str(last_build.get_status()))
Don't know if this was available at the time of the question, but jenkinsapi module's Job.invoke() and/or Jenkins.build_job() return a QueueItem object, which can block_until_building(), or block_until_complete()
jobq = server.build_job(job_name, job_params)
jobq.block_until_building()
print("Job %s (%s) is building." % (jobq.get_job_name(), jobq.get_build_number()))
jobq.block_until_complete(5) # check every 5s instead of the default 15
print("Job complete, %s" % jobq.get_build().get_status())
Was going through the same problem and this worked for me, using python3 and python-jenkins.
while "".join([d['color'] for d in j.get_jobs() if d['name'] == "job_name"]) == 'blue_anime':
print('Job is Running')
time.sleep(1)
print('Job Over!!')
Working Github Script: Link
This is working for me
#!/usr/bin/env python
import jenkins
import time
server = jenkins.Jenkins('https://jenkinsurl/', username='xxxxx', password='xxxxxx')
j_name = 'test'
server.build_job(j_name, {'testparam1': 'test', 'testparam2': 'test'})
while True:
print('Running....')
if server.get_job_info(j_name)['lastCompletedBuild']['number'] == server.get_job_info(j_name)['lastBuild']['number']:
print "Last ID %s, Current ID %s" % (server.get_job_info(j_name)['lastCompletedBuild']['number'], server.get_job_info(j_name)['lastBuild']['number'])
break
time.sleep(3)
print('Stop....')
console_output = server.get_build_console_output(j_name, server.get_job_info(j_name)['lastBuild']['number'])
print console_output
the issue main issue that the build_job doesn't return the number of the job, returns the number of a queue item (that only last 5 min). so the trick is
build_job
get the queue number,
with the queue number get the job_number
now we know the name of the job and the job number
get_job_info and loop the jobs till we find one with our job number
check the status
so i made a function for it with time_out
import time
from datetime import datetime, timedelta
import jenkins
def launch_job(jenkins_connection, job_name, parameters={}, wait=False, interval=30, time_out=7200):
"""
Create a jenkins job and waits for the job to finish
:param jenkins_connection: jenkins server jenkins object
:param job_name: the name of job we want to create and see if finish string
:param parameters: the parameters of the job to build directory
:param wait: if we want to wait for the job to finish or not bool
:param interval: how often we want to monitor seconds int
:param time_out: break the loop after certain X seconds int
:return: build job number int
"""
# we lunch the job and returns a queue_id
job_id = jenkins_connection.build_job(job_name, parameters)
# from the queue_id we get the job number that was created
queue_job = jenkins_connection.get_queue_item(job_id, depth=0)
build_number = queue_job["executable"]["number"]
print(f"job_name: {job_name} build_number: {build_number}")
if wait is True:
now = datetime.now()
later = now + timedelta(seconds=time_out)
while True:
# we check current time vs the timeout(later)
if datetime.now() > later:
raise ValueError(f"Job: {job_name}:{build_number} is running for more than {time_out} we"
f"stop monitoring the job, you can check it in Jenkins")
b = jenkins_connection.get_job_info(job_name, depth=1, fetch_all_builds=False)
for i in b["builds"]:
loop_id = i["id"]
if int(loop_id) == build_number:
result = (i["result"])
print(f"result: {result}") # in the json looks like null
if result is not None:
return i
# break
time.sleep(interval)
# return result
return build_number
after we ask jenkins to build the job>get queue#>get job#> loop the info and get the status till change from None to something else.
if works will return the directory with the information of that job. (hope the jenkins library could implement something like this.)
I have some code written in django/python. The principal is that the HTTP Response is a generator function. It spits the output of a subprocess on the browser window line by line. This works really well when I am using the django test server. When I use the real server it fails / basically it just beachballs when you press submit on the page before.
#condition(etag_func=None)
def pushviablah(request):
if 'hostname' in request.POST and request.POST['hostname']:
hostname = request.POST['hostname']
command = "blah.pl --host " + host + " --noturn"
return HttpResponse( stream_response_generator( hostname, command ), mimetype='text/html')
def stream_response_generator( hostname, command ):
proc = subprocess.Popen(command.split(), 0, None, subprocess.PIPE, subprocess.PIPE, subprocess.PIPE )
yield "<pre>"
var = 1
while (var == 1):
for line in proc.stdout.readline():
yield line
Anyone have any suggestions on how to get this working with on the real server? Or even how to debug why it is not working?
I discovered that the generator function is actually running but it has to complete before the httpresponse throws up a page onscreen. I don't want to have to wait for it to complete before the user sees output. I would like the user to see output as the subprocess progresses.
I'm wondering if this issue could be related to something in apache2 rather than django.
#evolution did you use gunicorn to deploy your app. If yes then you have created a service. I am having a similar kind of issue but with libreoffice. As much as I have researched I have found that PATH is overriding the command path present on your subprocess. I did not have a solution till now. If you bind your app with gunicorn in terminal then your code will also work.