I created a kvm domain,but it always crash after running about a day.
Here are some error messages in libvirtd.log:
2020-08-13 13:46:15.906+0000: 2018: error : qemuMonitorIO:718 : internal error：End of file from qemu monitor
2020-08-13 13:46:15.906+0000: 2018: debug : qemuMonitorIO:765 : Error on monitor internal error：End of file from qemu monitor
and some error messages in the domain's log:
qemu: qemu_thread_create: Resource temporarily unavailable
I have checked the free ram is enough
Run ulimit -a:
[root#localhost ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 385903
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 204800
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 1024
cpu time (seconds, -t) unlimited
max user processes (-u) 204800
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Related
we are using Ubuntu 16 LTS as a webserver with apache2.4, 18GB RAM and 20 cpus. I have issued following command
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 72012
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 72012
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Apache Version
$ apache2 -v
Server version: Apache/2.4.18 (Ubuntu)
Server built: 2018-04-18T14:53:04
Apache Module current in use
$ a2query -M
prefork
and MPM prefork module is used with default settings
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 5
MaxSpareServers 10
MaxRequestWorkers 400
MaxConnectionsPerChild 0
</IfModule>
I haved checked the maximum apache processes by JMeter stress tester and it doesnot expand more than 1500 processes. It is also checked that when 1500 processes are reached the RAM has still plenty of space (i.e. 9GB still remains).
Process count is identified by following command
ps -ef | grep apache2 | wc -l
Guidance required to increase that 1500 number as we need to stick with MPM_prefork module.
Try to change the ServerLimit in your config, see here.
You can also consider having a look at apache2buddy for guidance on tweaking your configuration.
Side note
Note that there is a hard limit of 15000.
Extract from Apache's documentation here
There is a hard limit of ThreadLimit 20000 (or ThreadLimit 100000 with event, ThreadLimit 15000 with mpm_winnt) compiled into the server. This is intended to avoid nasty effects caused by typos. To increase it even further past this limit, you will need to modify the value of MAX_THREAD_LIMIT in the mpm source file and rebuild the server.
I'm using Zaphoyd Websocketpp to creat a websocket server that need to accept very high concurrent connections ( C1M at least) on CentOS.
But the server process always get killed by kernel when the number of connections reached about 63k.
I see this message in dmesg:
Out of memory: Kill process 5420 (echo_server) score 382 or sacrifice child
Killed process 5420, UID 10545, (echo_server) total-vm:1488192kB, anon-rss:1467524kB, file-rss:32kB
I don't think the kernel will kill the process that only consumes about 1.5GB. So I created a simple program that allocates memory and do some read/write operations. This program was not killed by kernel. It only gets bad_alloc error when memory usage reaches 3.2GB.
I also checked some other parameters but found nothing suspicious:
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 29712
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1000000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 29712
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ cat /proc/sys/fs/nr_open
10485760
$ cat /proc/sys/fs/file-max
1280000
$ cat /proc/sys/fs/file-nr
1536 0 1280000
Can anyone help on this?
Are those connections created from the same machine?
What is the theoretical maximum number of open TCP connections that a modern Linux box can have
If a client has many connections to the same port on the same
destination, then three of those fields will be the same - only
source_port varies to differentiate the different connections. Ports
are 16-bit numbers, therefore the maximum number of connections any
given client can have to any given host port is 64K.
My program terminates with this error generating a core. The local disk wasnt full when I got this error. Error takes place from write() function.
My linux machine details are as follows,
Linux 2.6.18-274.18.1.el5 x86_64
I checked the ulimit -a and the details are as follows
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 49152
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 49152
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Nabla is correct. The block size of 1KB which was causing the problem. Thanks for your help.
In a C++ program (Linux), I need to write some numbers (integers, one number per line) to a file, and the size may be very large (currently 25GB).
The numbers are 1 , -1 or 0, which are used to record the connections of node and arc in a large graph.
all the output is written to a file (.txt) by std::ofstream << ...
The printing code architecture is :
for loop1 (node size)
for loop2 (arc size)
filename << .......
If the output size is small, it works well.
But, when the output size is large, the shell terminal where the programming is running is frozen.
But the process is still running and after long time (hours) it is killed by OS.
No errors, warnings, segmentation faluts pop up.
What are the possible reasons ?
I tried to search it online, but do not find what I need.
Thanks
This is the output of ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 399360
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 399360
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
in top command:
It used < 200 MB .
It is possible that your program has a memory leak (you said your program was large and complex), which would continually request more memory from the OS as your program runs. This could explain why your machine becomes unresponsive (due to memory pressure load), and also could explain why the OS terminates your program when it runs out of memory to give you.
Try watching your program run with top or something. If the resident size increases without bound, this may be your problem. With a smaller data set, you probably wouldn't notice the problem.
Here is the output.
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 104448
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16320
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I want to have a regex that matches all the (blocks, -c), (kbytes, -d), (-e) and so on... from each line.
You could try a regex like the following:
\([a-z\s\d,-]+\)