Ns2 nodes do not make concurrent http requests - if-statement

I wrote an ns2 script to run concurrent HTTP requests from 40 nodes out of 80 nodes.
However, the simulation showed one node, rather than 40 nodes, make requests to the HTTP server.
Could someone please clarify what might be wrong with the code?
The code is provided below
Thank you for your time, everyone.
set ns [new Simulator]
set clswitch [$ns node] # the access layer switch
set distrswitch [$ns node] # the distribution switch
set cch [$ns node] # the cache
set websrv [$ns node] # the web server
set traceFile [open Asim.tr w]
set namTraceFile [open Asim.nam w]
$ns namtrace-all $namTraceFile
$ns trace-all $traceFile
proc finish {} {
global ns traceFile namTraceFile
$ns flush-trace-all
puts "Simulation completed."
flush $traceFile
flush $namTraceFile
close $traceFile
close $namTraceFile
exit 0
}
for {set i 0} {$i < 80} {incr i} {
if {$i % 2 == 0} {
set cl ($i) [$ns node]
$ns duplex-link $cl($i) $clswitch 100Mb 10ms DropTail
set tcpAgent [new Agent/TCP]
$ns attach-agent $cl($i) $tcpAgent
set tcpSink [new Agent/TCPSink]
$ns attach-agent $clswitch $tcpSink
$ns connect $tcpAgent $tcpSink
set client [new Http/Client $ns $cl($i)]
proc start-connection {} {
global ns client cache server
$client connect $cache
$cache connect $server
$client start-session $cache $server
}
$ns at 0.5 "start-connection"
}
}
$ns duplex-link $clswitch $distrswitch 100Mb 10ms DropTail
set tcpAgent [new Agent/TCP]
$ns attach-agent $clswitch $tcpAgent
set tcpSink [new Agent/TCPSink]
$ns attach-agent $distrswitch $tcpSink
$ns connect $tcpAgent $tcpSink
$ns duplex-link $distrswitch $cch 100Mb 10ms DropTail
set tcpAgent [new Agent/TCP]
$ns attach-agent $distrswitch $tcpAgent
set tcpSink [new Agent/TCPSink]
$ns attach-agent $cch $tcpSink
$ns connect $tcpAgent $tcpSink
$ns duplex-link $cch $websrv 100Mb 10ms DropTail
set tcpAgent [new Agent/TCP]
$ns attach-agent $cch $tcpAgent
set tcpSink [new Agent/TCPSink]
$ns attach-agent $websrv $tcpSink
$ns connect $tcpAgent $tcpSink
set server [new Http/Server $ns $websrv]
set cache [new Http/Cache $ns $cch]
set pgp [new PagePool/Math]
set tmp [new RandomVariable/Constant]
$tmp set val_ 4096
$pgp ranvar-size $tmp
set tmp [new RandomVariable/Exponential]
$tmp set avg_ 6
$pgp ranvar-age $tmp
$server set-page-generator $pgp
set tmp [new RandomVariable/Exponential]
$tmp set avg_ 0.5
$client set-interval-generator $tmp
$client set-page-generator $pgp
$ns at 6.0 "finish"

The problem is in this code:
for {set i 0} {$i < 80} {incr i} {
if {$i % 2 == 0} {
set cl ($i) [$ns node]
$ns duplex-link $cl($i) $clswitch 100Mb 10ms DropTail
set tcpAgent [new Agent/TCP]
$ns attach-agent $cl($i) $tcpAgent
set tcpSink [new Agent/TCPSink]
$ns attach-agent $clswitch $tcpSink
$ns connect $tcpAgent $tcpSink
set client [new Http/Client $ns $cl($i)]
proc start-connection {} {
global ns client cache server
$client connect $cache
$cache connect $server
$client start-session $cache $server
}
$ns at 0.5 "start-connection"
}
}
First off, you're defining a procedure inside a loop. That's a bad indication right there (since you're not doing complex runtime code generation). But the real problem is that while you're creating many clients, you're trying to store them all in the same simple variable:
set client [new Http/Client $ns $cl($i)]
Each time through the loop, this gets overwritten; only the last such client will get started. What we need to do is to pass the client to the procedure by an argument.
##### MOVED, AND client IS NOW AN ARGUMENT, NOT A GLOBAL #####
proc start-connection {client} {
global ns cache server
$client connect $cache
$cache connect $server
$client start-session $cache $server
}
##### Now same as above... except for a couple of lines #####
for {set i 0} {$i < 80} {incr i} {
if {$i % 2 == 0} {
set cl ($i) [$ns node]
$ns duplex-link $cl($i) $clswitch 100Mb 10ms DropTail
set tcpAgent [new Agent/TCP]
$ns attach-agent $cl($i) $tcpAgent
set tcpSink [new Agent/TCPSink]
$ns attach-agent $clswitch $tcpSink
$ns connect $tcpAgent $tcpSink
set client [new Http/Client $ns $cl($i)]
##### THE NEXT LINE IS CHANGED FROM YOUR CODE! #####
$ns at 0.5 [list start-connection $client]
##### KEEP TRACK OF ALL THE CLIENTS #####
lappend allClients $client
}
}
We also need to do a little work later on, where you refer to the “client” in some lower code. The work just needs to be done in a loop iterating over the list of clients.
foreach client $allClients {
$client set-interval-generator $tmp
$client set-page-generator $pgp
}
Lists are a great way of keeping a simple collection of things.

Related

karma test cases are running multiple times: Angular unit test

I am not able to understand why my karma test cases re runs multiple times after a fixed test case.
Disconnected, because no message in 50000 ms.
Chrome 75.0.3770 (Windows 10.0.0): Executed 131 of 251 DISCONNECTED (1 min 9.028 secs / 18.285 secs)
Chrome 75.0.3770 (Windows 10.0.0): Executed 131 of 251 DISCONNECTED (47.273 secs / 18.169 secs)
Chrome 75.0.3770 (Windows 10.0.0): Executed 131 of 251 DISCONNECTED (1 min 9.028 secs / 18.285 secs)
Chrome 75.0.3770 (Windows 10.0.0): Executed 131 of 251 DISCONNECTED (47.273 secs / 18.169 secs)
Chrome 75.0.3770 (Windows 10.0.0): Executed 97 of 251 DISCONNECTED (22.07 secs / 19.87 secs)
Chrome 75.0.3770 (Windows 10.0.0): Executed 131 of 251 DISCONNECTED (1 min 9.028 secs / 18.285 secs)
Chrome 75.0.3770 (Windows 10.0.0): Executed 131 of 251 SUCCESS (0 secs / 17.406 secs)
Chrome 75.0.3770 (Windows 10.0.0): Executed 239 of 251 SUCCESS (0 secs / 20.912 secs)
Below is the configuration of karma.conf.js
// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html
const { join } = require('path');
const { constants } = require('karma');
module.exports = () => {
return {
basePath: '',
frameworks: ['jasmine', '#angular-devkit/build-angular'],
plugins: [
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-jasmine-html-reporter'),
require('karma-coverage-istanbul-reporter'),
require('#angular-devkit/build-angular/plugins/karma')
],
client: {
clearContext: false, // leave Jasmine Spec Runner output visible in browser
jasmine: {
random: false
}
},
coverageIstanbulReporter: {
dir: join(__dirname, '../../coverage'),
reports: ['html', 'lcovonly'],
fixWebpackSourcePaths: true
},
reporters: ['progress', 'kjhtml'],
port: 9876,
colors: true,
logLevel: constants.LOG_INFO,
autoWatch: true,
browsers: ['Chrome'],
singleRun: true,
};
};
when I change singleRun to false and execute test case using ng test, it works fine, the problem is only when I change it to true.
I saw couple of blogs and tried to resolve it by adding some more configuration like
browserNoActivityTimeout: 50000,
captureTimeout: 60 * 1000,
// captureTimeout: 210000,
browserDisconnectTolerance: 1,
// browserDisconnectTimeout: 210000,
// browserNoActivityTimeout: 210000
but nothing has worked for me.
Below is the configuration in devDependencies (I am using angular cli version 8)
"devDependencies": {
"#angular-devkit/build-angular": "0.800.3",
"#angular-devkit/build-ng-packagr": "0.800.3",
"#angular/cli": "^8.0.3",
"#angular/compiler-cli": "~8.0.0",
"#angular/language-service": "^6.1.0",
"#ngrx/store-devtools": "6.0.1",
"#nrwl/schematics": "~8.0.0",
"#ruf/schematics": "next",
"#ruf/wrench": "next",
"#types/jasmine": "~2.8.6",
"#types/jasminewd2": "~2.0.3",
"#types/node": "^8.10.10",
"codelyzer": "5.1.0",
"dotenv": "6.2.0",
"jasmine-core": "~3.4.0",
"jasmine-marbles": "0.3.1",
"jasmine-spec-reporter": "~4.2.1",
"karma": "~4.1.0",
"karma-chrome-launcher": "~2.2.0",
"karma-coverage-istanbul-reporter": "^2.0.5",
"karma-jasmine": "~2.0.1",
"karma-jasmine-html-reporter": "~1.4.2",
"ngrx-store-freeze": "0.2.4",
"prettier": "1.16.4",
"protractor": "~5.3.0",
"scss-bundle": "^2.1.2",
"ts-node": "~7.0.0",
"tslint": "~5.11.0",
"typescript": "~3.4.5",
"istanbul-instrumenter-loader": "^3.0.1",
"karma-sourcemap-loader": "^0.3.7",
"karma-webpack": "^3.0.4",
"source-map-loader": "^0.2.4"
}
Please suggest !!!!
I fixed the above issue by spying on functions which has location.href set inside it.
location.href was causing the page to load with different URL and because of that karma was running again and again.

Why doesn't this program copy all my files, using scp?

I wrote a little program which, in first case, slices a list of files and directory names so I get a list of files and a list of directories.
#!/usr/bin/expect -f
set mdproot "****"
set mySource "****"
set myDest "****"
set myDirs {}
set myFiles {}
set i 1
set y 0
lappend myDirs [lindex $argv 0]
foreach variable $argv {
lappend myDirs [lindex $argv $i+1]
lappend myFiles [lindex $argv $y+1]
incr y
incr y
incr i
incr i
}
set DIRECTORIES [lsearch -all -inline -not -exact $myDirs {}]
set FILES [lsearch -all -inline -not -exact $myFiles {}]
#puts " "
#puts $DIRECTORIES
#puts " "
#puts $FILES
foreach file $FILES dir $DIRECTORIES {
puts " Fichier : $file et repertoire : $dir"
spawn scp -p "$mySource/$file" "$myDest/$dir"
expect -re "(.*)assword: " {sleep 1; send -- "$mdproot\r" }
expect eof { return}
}
There are my lists:
$argv :
2017-11-30
2017-11-30_15-10-44_P_8294418.33_Q1
2017-11-30
2017-11-30_15-10-44_R_8294418.33_Q1
2018-03-07
2018-03-07_09-30-57_R_HOURS_Q1
2018-04-13
2018-04-13_13-23-25_R_HOURS_Q1
2018-05-02
2018-05-02_11-19-37_R_HOURS_Q1
2018-03-07
2018-3-7_9-30-57_P_HOURS_Q1
2018-04-13
2018-4-13_13-23-25_P_HOURS_Q1
2018-05-02
2018-5-2_11-19-37_P_HOURS_Q1
$DIRECTORIES :
2017-11-30
2017-11-30
2018-03-07
2018-04-13
2018-05-02
2018-03-07
2018-04-13
2018-05-02
$FILES :
2017-11-30_15-10-44_P_8294418.33_Q1
2017-11-30_15-10-44_R_8294418.33_Q1
2018-03-07_09-30-57_R_HOURS_Q1
2018-04-13_13-23-25_R_HOURS_Q1
2018-05-02_11-19-37_R_HOURS_Q1
2018-3-7_9-30-57_P_HOURS_Q1
2018-4-13_13-23-25_P_HOURS_Q1
2018-5-2_11-19-37_P_HOURS_Q1
Actually I have 2 problems (3 in the case we count how trash this code is).
First, when I run my program, I have my % indicator for each file I am copying and except the last one, they all stop before they get to 100%.
Then, I can see that the scp command isn't done on all the files, the program stops pretty much every time at the 4th file.
root#raspberrypi:~# ./recupFileName.sh
spawn scp -p /root/muonic_data/2017-11-30_15-10-44_P_8294418.33_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2017-11-30
marpic#192.168.110.90's password:
2017-11-30_15-10-44_P_8294418.33_Q1 15% 68MB 6.9MB/s 00:53
spawn scp -p /root/muonic_data/2017-11-30_15-10-44_R_8294418.33_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2017-11-30
marpic#192.168.110.90's password:
2017-11-30_15-10-44_R_8294418.33_Q1 41% 69MB 8.5MB/s 00:11
spawn scp -p /root/muonic_data/2018-03-07_09-30-57_R_HOURS_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2018-03-07
marpic#192.168.110.90's password:
2018-03-07_09-30-57_R_HOURS_Q1 82% 51MB 7.2MB/s 00:01
spawn scp -p /root/muonic_data/2018-04-13_13-23-25_R_HOURS_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2018-04-13
marpic#192.168.110.90's password:
2018-04-13_13-23-25_R_HOURS_Q1 100% 6940KB 6.8MB/s 00:01
As you can see, there should be 8 files copied with 100% accuracy, but there are no error messages so I don't know where to start my research.
EDIT :
I added the "set timeout -1" in my script, but now the script is copying only my first file with 100% accuracy, then stops. Any answers ?
root#raspberrypi:~# ./recupFileName.sh
Fichier : 2017-11-30_15-10-44_P_8294418.33_Q1 et repertoire : 2017-11-30
spawn scp -p /root/muonic_data/2017-11-30_15-10-44_P_8294418.33_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2017-11-30
marpic#192.168.110.90's password:
2017-11-30_15-10-44_P_8294418.33_Q1 100% 437MB 7.5MB/s 00:58
root#raspberrypi:~#
The problem should be in expect eof. By default the timeout is 10 seconds so expect eof would return after 10 seconds though the scp is still running.
You can use a larger timeout.
Option #1:
# set the default `timeout'
set timeout 3600 ; # or -1 for no timeout
Option #2:
expect -timeout 3600 eof
Note that your
expect eof { return }
would exit the whole script so the foreach loop runs for only once, you need just
expect eof

Tuning gunicorn (with Django): Optimize for more concurrent connections and faster connections

I use Django 1.5.3 with gunicorn 18.0 and lighttpd. I serve my static and dynamic content like that using lighttpd:
$HTTP["host"] == "www.mydomain.com" {
$HTTP["url"] !~ "^/media/|^/static/|^/apple-touch-icon(.*)$|^/favicon(.*)$|^/robots\.txt$" {
proxy.balance = "hash"
proxy.server = ( "" => ("myserver" =>
( "host" => "127.0.0.1", "port" => 8013 )
))
}
$HTTP["url"] =~ "^/media|^/static|^/apple-touch-icon(.*)$|^/favicon(.*)$|^/robots\.txt$" {
alias.url = (
"/media/admin/" => "/var/www/virtualenvs/mydomain/lib/python2.7/site-packages/django/contrib/admin/static/admin/",
"/media" => "/var/www/mydomain/mydomain/media",
"/static" => "/var/www/mydomain/mydomain/static"
)
}
url.rewrite-once = (
"^/apple-touch-icon(.*)$" => "/media/img/apple-touch-icon$1",
"^/favicon(.*)$" => "/media/img/favicon$1",
"^/robots\.txt$" => "/media/robots.txt"
)
}
I already tried to run gunicorn (via supervisord) in many different ways, but I cant get it better optimized than it can handle about 1100 concurrent connections. In my project I need about 10000-15000 connections
command = /var/www/virtualenvs/myproject/bin/python /var/www/myproject/manage.py run_gunicorn -b 127.0.0.1:8013 -w 9 -k gevent --preload --settings=myproject.settings
command = /var/www/virtualenvs/myproject/bin/python /var/www/myproject/manage.py run_gunicorn -b 127.0.0.1:8013 -w 10 -k eventlet --worker_connections=1000 --settings=myproject.settings --max-requests=10000
command = /var/www/virtualenvs/myproject/bin/python /var/www/myproject/manage.py run_gunicorn -b 127.0.0.1:8013 -w 20 -k gevent --settings=myproject.settings --max-requests=1000
command = /var/www/virtualenvs/myproject/bin/python /var/www/myproject/manage.py run_gunicorn -b 127.0.0.1:8013 -w 40 --settings=myproject.settings
On the same server there live about 10 other projects, but CPU and RAM is fine, so this shouldnt be a problem, right?
I ran a load test and these are the results:
At about 1100 connections my lighttpd errorlog says something like that, thats where the load test shows the drop of connections:
2013-10-31 14:06:51: (mod_proxy.c.853) write failed: Connection timed out 110
2013-10-31 14:06:51: (mod_proxy.c.939) proxy-server disabled: 127.0.0.1 8013 83
2013-10-31 14:06:51: (mod_proxy.c.1316) no proxy-handler found for: /
... after about one minute
2013-10-31 14:07:02: (mod_proxy.c.1361) proxy - re-enabled: 127.0.0.1 8013
These things also appear ever now and then:
2013-10-31 14:06:55: (network_linux_sendfile.c.94) writev failed: Connection timed out 600
2013-10-31 14:06:55: (mod_proxy.c.853) write failed: Connection timed out 110
...
2013-10-31 14:06:57: (mod_proxy.c.828) establishing connection failed: Connection timed out
2013-10-31 14:06:57: (mod_proxy.c.939) proxy-server disabled: 127.0.0.1 8013 45
So how can I tune gunicorn/lighttpd to serve more connections faster? What can I optimize? Do you know any other/better setup?
Thanks alot in advance for your help!
Update: Some more server info
root#django ~ # top
top - 15:28:38 up 100 days, 9:56, 1 user, load average: 0.11, 0.37, 0.76
Tasks: 352 total, 1 running, 351 sleeping, 0 stopped, 0 zombie
Cpu(s): 33.0%us, 1.6%sy, 0.0%ni, 64.2%id, 0.4%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 32926156k total, 17815984k used, 15110172k free, 342096k buffers
Swap: 23067560k total, 0k used, 23067560k free, 4868036k cached
root#django ~ # iostat
Linux 2.6.32-5-amd64 (django.myserver.com) 10/31/2013 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
33.00 0.00 2.36 0.40 0.00 64.24
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 137.76 980.27 2109.21 119567783 257268738
sdb 24.23 983.53 2112.25 119965731 257639874
sdc 24.25 985.79 2110.14 120241256 257382998
md0 0.00 0.00 0.00 400 0
md1 0.00 0.00 0.00 284 6
md2 1051.93 38.93 4203.96 4748629 512773952
root#django ~ # netstat -an |grep :80 |wc -l
7129
Kernel Settings:
echo "10152 65535" > /proc/sys/net/ipv4/ip_local_port_range
sysctl -w fs.file-max=128000
sysctl -w net.ipv4.tcp_keepalive_time=300
sysctl -w net.core.somaxconn=250000
sysctl -w net.ipv4.tcp_max_syn_backlog=2500
sysctl -w net.core.netdev_max_backlog=2500
ulimit -n 10240

Google datetime formatter doesn't work for Google Line Chart

For some reason, no matter what I do and how I try, this date formatter just doesn't work. The tooltip is still showing the wrong format. You can try the code here.
In the code I have to comment out formatter.format(dataTable, 0) otherwise it does not work.
Is there a way to let the tooltip show the correct format?
function drawVisualization() {
// Create and populate the data table.
var dataTable= new google.visualization.DataTable();
dataTable.addColumn('datetime', 'Year');
dataTable.addColumn('number', 'Apple');
dataTable.addColumn('number', 'Android');
dataTable.addRows([
[new Date("6/22/2009 16:00:00"),425,45 ],
[new Date("6/8/2009 16:00:00"),424, 43],
[new Date("5/26/2009 16:00:00"),417, 434],
[new Date("5/11/2009 16:00:00"),390, 343],
[new Date("4/27/2009 16:00:00"),393, 45],
[new Date("4/13/2009 16:00:00"),392, 45],
[new Date("3/30/2009 16:00:00"),369, 868],
[new Date("3/16/2009 16:00:00"),330, 68],
[new Date("3/2/2009 16:00:00"),308, 86],
[new Date("2/17/2009 16:00:00"),346, 43],
[new Date("2/2/2009 16:00:00"),371, 38],
[new Date("1/20/2009 16:00:00"),324, 86],
[new Date("1/5/2009 16:00:00"),315, 78],
[new Date("12/22/2008 16:00:00"),300, 536],
[new Date("12/8/2008 16:00:00"),315, 38],
[new Date("11/24/2008 16:00:00"),292, 78],
[new Date("11/10/2008 16:00:00"),310, 99],
[new Date("10/27/2008 16:00:00"),359, 78],
[new Date("10/13/2008 16:00:00"),372, 68],
[new Date("9/29/2008 16:00:00"),386, 68],
[new Date("9/15/2008 16:00:00"),449, 78],
[new Date("9/2/2008 16:00:00"),444, 69],
[new Date("8/25/2008 16:00:00"),463, 25],
[new Date("8/11/2008 16:00:00"),510, 53],
[new Date("7/28/2008 16:00:00"),467, 38],
[new Date("7/14/2008 16:00:00"),481, 45],
[new Date("6/30/2008 16:00:00"),537, 85],
[new Date("6/16/2008 16:00:00"),546, 123],
[new Date("6/2/2008 16:00:00"),567, 234],
[new Date("5/19/2008 16:00:00"),544, 45],
[new Date("5/5/2008 16:00:00"),573, 54],
[new Date("4/21/2008 16:00:00"),544, 88],
[new Date("4/7/2008 16:00:00"),457, 86],
[new Date("3/24/2008 16:00:00"),438, 78],
[new Date("3/10/2008 16:00:00"),437, 68],
[new Date("2/25/2008 16:00:00"),471, 35],
[new Date("2/11/2008 16:00:00"),529, 45],
[new Date("1/28/2008 16:00:00"),515, 47],
[new Date("1/14/2008 16:00:00"),600, 57],
[new Date("12/31/2007 16:00:00"),657, 57],
[new Date("12/17/2007 16:00:00"),696, 83],
[new Date("12/3/2007 16:00:00"),714, 387],
[new Date("11/19/2007 16:00:00"),676, 75],
[new Date("11/5/2007 16:00:00"),663, 85],
[new Date("10/22/2007 16:00:00"),674, 733],
[new Date("10/8/2007 16:00:00"),637, 327],
[new Date("9/24/2007 16:00:00"),567, 345],
[new Date("9/10/2007 16:00:00"),528, 45],
[new Date("8/27/2007 16:00:00"),515, 324]]);
// For some reason, it doesn't work correctly
var formatter = google.visualization.DateFormat({pattern: 'yyyy/MM/dd'});
//formatter.format(dataTable, 0);
// Create and draw the visualization.
var chart = new google.visualization.LineChart(document.getElementById('visualization'));
chart.draw(dataTable, {curveType: "function"});
}
You have a syntax error in your format code.
Wrong:
var formatter = google.visualization.DateFormat({pattern: 'yyyy/MM/dd'});
Right:
var formatter = new google.visualization.DateFormat({pattern: 'yyyy/MM/dd'});
You need to add the 'new' prior to the DateFormat argument. This single change makes your code work properly.

Celeryd Worker stops processing tasks

I have a celeryd running with two workers, and watching them I see them accept 4 tasks each, process them, and then stop processing tasks. (Note that these tasks are long running, up to 2mins to process).
celeryctl provides the following information:
django#server: ./manage.py celeryctl inspect active
<- active
DEBUG 2012-06-01 12:51:11,330 amqplib 661 Start from server, version: 8.0, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2012 VMware, Inc.', u'capabilities': {}, u'platform': u'Erlang/OTP', u'version': u'2.8.2'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
DEBUG 2012-06-01 12:51:11,331 amqplib 507 Open OK! known_hosts []
DEBUG 2012-06-01 12:51:11,331 amqplib 70 using channel_id: 1
DEBUG 2012-06-01 12:51:11,332 amqplib 484 Channel open
-> eso-dev: OK
- empty -
django#server: ./manage.py celeryctl inspect scheduled
<- scheduled
DEBUG 2012-06-01 12:52:07,107 amqplib 661 Start from server, version: 8.0, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2012 VMware, Inc.', u'capabilities': {}, u'platform': u'Erlang/OTP', u'version': u'2.8.2'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
DEBUG 2012-06-01 12:52:07,108 amqplib 507 Open OK! known_hosts []
DEBUG 2012-06-01 12:52:07,108 amqplib 70 using channel_id: 1
DEBUG 2012-06-01 12:52:07,109 amqplib 484 Channel open
-> eso-dev: OK
- empty -
django#server: ./manage.py celeryctl inspect registered
<- registered
DEBUG 2012-06-01 12:52:20,567 amqplib 661 Start from server, version: 8.0, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2012 VMware, Inc.', u'capabilities': {}, u'platform': u'Erlang/OTP', u'version': u'2.8.2'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
DEBUG 2012-06-01 12:52:20,568 amqplib 507 Open OK! known_hosts []
DEBUG 2012-06-01 12:52:20,568 amqplib 70 using channel_id: 1
DEBUG 2012-06-01 12:52:20,569 amqplib 484 Channel open
-> eso-dev: OK
* celery.backend_cleanup
* celery.chord
* celery.chord_unlock
*
celery_haystack.tasks.CeleryHaystackSignalHandler
*
celery_haystack.tasks.CeleryHaystackUpdateIndex
* convert.tasks.create_pdf
* convert.tasks.create_pngs
In addition every time this happens the last thing to be printed to logs is:
[2012-06-01 12:17:53,777: INFO/MainProcess] Task convert.tasks.create_pdf[319984de-5bc4-47fc-891f-273d827d625f] retry: None
None
[2012-06-01 12:17:54,327: INFO/MainProcess] Task convert.tasks.create_pdf[8a89f3c1-e991-487e-a2db-a57d23bae17f] retry: None
None
The tasks also happen to have failed just before this is printed, and in my code all I have called is
:
except HTTPError, e:
statsd.incr('A.stat')
log.warn('Woops: %s', e)
create_pdf.retry()
If I kill celeryd (^C, and it dies straight away, no waiting for tasks) and start it again it continues like nothing has happend for a few more tasks and dies again (I think its always on the create_pdf task, but the logs show this task failing and being retried without a problem)
This is a know bug with celery that was fixed in v3.0. (Issue 707)