Cleaning up disk space on Centos OS - web-services

I'm having issues with inconsistently my site going down (giving a 500 error) and when running commands getting:
-jailshell: fork: retry: Resource temporarily unavailable
I'm a noob on the server.
I ran df -h and I got back:
Filesystem Size Used Avail Use% Mounted on
/dev/root 39G 24G 14G 65% /usr/local/cpanel/3rdparty/mailman
/dev/root 152G 102G 49G 68% /var/spool
/dev/root 39G 24G 14G 65% /usr/sbin
/dev/root 15G 9.3G 4.3G 69% /etc/mail
/dev/root 9.5G 1.3G 7.8G 15% /var/tmp
/dev/root 15G 9.3G 4.3G 69% /lib64
/dev/root 15G 9.3G 4.3G 69% /sbin
/dev/root 15G 9.3G 4.3G 69% /opt
/dev/root 39G 24G 14G 65% /usr
/dev/root 152G 102G 49G 68% /var
/dev/root 9.5G 1.3G 7.8G 15% /tmp
/dev/root 32G 164K 32G 1% /dev
/dev/root 15G 9.3G 4.3G 69% /bin
/dev/root 15G 9.3G 4.3G 69% /lib
/dev/root 1.8T 1.2T 639G 65% /home4/sitedir
sitedir being where everything is hosted.
I feel that 1.2T of mem being used is a lot. But i'm unsure. Is there anyway I can look at what's causing this amount of memory in /home4/sitedir and possibly how I could clean it up?

For starters, try running the top command as root and press the m key to sort the processes by memory once top is running. You should see the processes using the most memory since the m key sorts by memory usage.
As I mentioned in my comment above, the question asks about disk space but then refers to memory in the body. You have a command to give you the disk space ( df -h ) so I am giving you a response for the (possible) memory side of the your question, if indeed that is what you are concerned about.

Related

Abnormally large drivers/modules directories when building a custom kernel on Ubuntu 18.04 / 20.04

I'm currently tiptoeing into custom kernel building.
I first started on a VM in VirtualBox, installing a fresh distribution of Ubuntu Server 20.04.
I followed the following procedure:
# getting the archive
cd ~/src/linux/
wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.91.tar.xz
tar -xf linux-5.4.91.tar.xz
cd linux-5.4.91/
# getting the config of the kernel currently running
cp -v /boot/config-$(uname -r) .config
# adapting the config to the target kernel, selecting the default for everything
make oldconfig
# adding manually a suffix to the kernel name: "-tony"
make menuconfig
# building and installing modules & image
time make -j $(nproc)
sudo make modules_install
sudo make install
Everything goes fine, although the build takes a bit longer than expected (~30 minutes on a 6-core AMD Ryzen 5 3600 6-Core Processor, with 32 GB of RAM; I assigned 6 CPUs over the 12 available to the VM).
However, when I check the size of the source directory, now containing the object files of everything built, the size of the drivers directory bothers me:
user#vm:~/src/linux/linux-5.4.91$ du -hd 1
1.3G ./fs
544K ./certs
34M ./block
128K ./usr
41M ./tools
208K ./LICENSES
58M ./security
48M ./Documentation
3.4M ./init
58M ./include
895M ./sound
73M ./lib
5.1M ./ipc
95M ./crypto
2.0G ./net
2.2M ./samples
446M ./arch
5.2M ./scripts
195M ./.git
4.8M ./virt
142M ./kernel
56M ./mm
13G ./drivers
20G .
13 GB looks like a lot.
When I look at the installed files, the modules also looks very big (5.6G):
root#vm:/# find -name '*5.4.91*' | xargs du -hs
20G ./home/user/src/linux/linux-5.4.91
105M ./home/user/src/linux/linux-5.4.91.tar.xz
5.6G ./usr/lib/modules/5.4.91-tony+
4.0K ./var/lib/initramfs-tools/5.4.91-tony+
912M ./boot/initrd.img-5.4.91-tony+
232K ./boot/config-5.4.91-tony+
12M ./boot/vmlinuz-5.4.91-tony+
4.5M ./boot/System.map-5.4.91-tony+
Especially when I compare with the vanilla kernel installed with the distribution:
root#vm:/# find -name '*5.4.0-62*' | xargs du -hs
...
262M ./usr/lib/modules/5.4.0-62-generic
4.0K ./usr/lib/modprobe.d/blacklist_linux_5.4.0-62-generic.conf
4.0K ./var/lib/initramfs-tools/5.4.0-62-generic
...
12M ./boot/vmlinuz-5.4.0-62-generic
79M ./boot/initrd.img-5.4.0-62-generic
236K ./boot/config-5.4.0-62-generic
4.6M ./boot/System.map-5.4.0-62-generic
262M vs 5.6G seems like a lot of difference, taken into account that I took the same config file (the cp -v /boot/config-$(uname -r) .config command).
I also reproduced the same results on Ubuntu Server 18.04.
Moreover, I tried on my host directly (no VM) with again the same results.
I am obviously missing something here, but I cannot find what:
should I strip the kernel / the modules from unused symbols?
is there a configuration that is missing somewhere ?
Thank you in advance for your help!
Disabling CONFIG_SLUB_DEBUG, CONFIG_DEBUG_INFO and CONFIG_DEBUG_MISC in the kernel configuration resulted in reasonable build performance:
real 22m19.559s
user 119m17.273s
sys 12m15.424s
And the following size for the modules:
root#vm:/usr/lib/modules# du -hs *
261M 5.10.10-stripped+
262M 5.4.0-64-generic
Which is even better than the shipped kernel :) (notice I updated the kernel source, but it doesn't change the result if using the same kernel).
Apparently, one can also use make modules_install INSTALL_MOD_STRIP=1 to strip the modules when they are built with debug symbols, reducing the size without having to change the build options.
Installing with that option on the already-stripped kernel & modules as described above resulted in the following size:
user#vm:/usr/lib/modules$ du -hs *
260M 5.10.10-stripped+
262M 5.4.0-64-generic
We effectively gain one more megabyte from this additionnal stripping.

apt-get -f install getting disk space error in-spite of space [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 years ago.
Improve this question
I want to set cloudwatch alerts on my production server but getting dependency errors for libwww-perl while resolving it is saying "No space left on device" though there is space. please let me know what is that I am missing.
ubuntu#ip-172-21-0-17:/opt/aws-scripts-mon$ ./mon-put-instance-data.pl
Can't locate URI/Escape.pm in #INC (you may need to install the URI::Escape module) (#INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.22.1 /usr/local/share/perl/5.22.1 /usr/lib/x86_64-linux-gnu/perl5/5.22 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.22 /usr/share/perl/5.22 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base . .) at AwsSignatureV4.pm line 20.
BEGIN failed--compilation aborted at AwsSignatureV4.pm line 20.
Compilation failed in require at CloudWatchClient.pm line 21.
BEGIN failed--compilation aborted at CloudWatchClient.pm line 21.
Compilation failed in require at ./mon-put-instance-data.pl line 86.
BEGIN failed--compilation aborted at ./mon-put-instance-data.pl line 86.
ubuntu#ip-172-21-0-17:/opt/aws-scripts-mon$ sudo apt-get install libwww-perl libdatetime-perl -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
libdatetime-perl : Depends: libdatetime-locale-perl but it is not going to be installed
Depends: libdatetime-timezone-perl (>= 1:1.74) but it is not going to be installed
Depends: libparams-validate-perl (>= 1.03) but it is not going to be installed
Depends: libtry-tiny-perl but it is not going to be installed
libwww-perl : Depends: libencode-locale-perl but it is not going to be installed
Depends: libfile-listing-perl but it is not going to be installed
Depends: libhtml-parser-perl but it is not going to be installed
Depends: libhtml-tagset-perl but it is not going to be installed
Depends: libhtml-tree-perl but it is not going to be installed
Depends: libhttp-cookies-perl but it is not going to be installed
Depends: libhttp-date-perl but it is not going to be installed
Depends: libhttp-message-perl but it is not going to be installed
Depends: libhttp-negotiate-perl but it is not going to be installed
Depends: liblwp-mediatypes-perl but it is not going to be installed
Depends: liblwp-protocol-https-perl but it is not going to be installed
Depends: libnet-http-perl (>= 6.07) but it is not going to be installed
Depends: liburi-perl but it is not going to be installed
Depends: libwww-robotrules-perl but it is not going to be installed
Recommends: libhtml-form-perl but it is not going to be installed
Recommends: libhtml-format-perl but it is not going to be installed
Recommends: libhttp-daemon-perl but it is not going to be installed
Recommends: libmailtools-perl but it is not going to be installed
linux-headers-4.4.0-157-generic : Depends: linux-headers-4.4.0-157 but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
ubuntu#ip-172-21-0-17:/opt/aws-scripts-mon$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
linux-headers-4.4.0-101 linux-headers-4.4.0-101-generic linux-headers-4.4.0-103 linux-headers-4.4.0-103-generic linux-headers-4.4.0-104 linux-headers-4.4.0-104-generic linux-headers-4.4.0-108
linux-headers-4.4.0-108-generic linux-headers-4.4.0-109 linux-headers-4.4.0-109-generic linux-headers-4.4.0-112 linux-headers-4.4.0-112-generic linux-headers-4.4.0-116
linux-headers-4.4.0-116-generic linux-headers-4.4.0-119 linux-headers-4.4.0-119-generic linux-headers-4.4.0-121 linux-headers-4.4.0-121-generic linux-headers-4.4.0-124
linux-headers-4.4.0-124-generic linux-headers-4.4.0-127 linux-headers-4.4.0-127-generic linux-headers-4.4.0-128 linux-headers-4.4.0-128-generic linux-headers-4.4.0-130
linux-headers-4.4.0-130-generic linux-headers-4.4.0-133 linux-headers-4.4.0-133-generic linux-headers-4.4.0-134 linux-headers-4.4.0-134-generic linux-headers-4.4.0-137
linux-headers-4.4.0-137-generic linux-headers-4.4.0-138 linux-headers-4.4.0-138-generic linux-headers-4.4.0-139 linux-headers-4.4.0-139-generic linux-headers-4.4.0-141
linux-headers-4.4.0-141-generic linux-headers-4.4.0-142 linux-headers-4.4.0-142-generic linux-headers-4.4.0-143 linux-headers-4.4.0-143-generic linux-headers-4.4.0-148
linux-headers-4.4.0-148-generic linux-headers-4.4.0-150 linux-headers-4.4.0-150-generic linux-headers-4.4.0-59 linux-headers-4.4.0-59-generic linux-headers-4.4.0-66
linux-headers-4.4.0-66-generic linux-headers-4.4.0-70 linux-headers-4.4.0-70-generic linux-headers-4.4.0-71 linux-headers-4.4.0-71-generic linux-headers-4.4.0-72 linux-headers-4.4.0-72-generic
linux-headers-4.4.0-75 linux-headers-4.4.0-75-generic linux-headers-4.4.0-78 linux-headers-4.4.0-78-generic linux-headers-4.4.0-79 linux-headers-4.4.0-79-generic linux-headers-4.4.0-81
linux-headers-4.4.0-81-generic linux-headers-4.4.0-83 linux-headers-4.4.0-83-generic linux-headers-4.4.0-87 linux-headers-4.4.0-87-generic linux-headers-4.4.0-89 linux-headers-4.4.0-89-generic
linux-headers-4.4.0-91 linux-headers-4.4.0-91-generic linux-headers-4.4.0-92 linux-headers-4.4.0-92-generic linux-headers-4.4.0-93 linux-headers-4.4.0-93-generic linux-headers-4.4.0-96
linux-headers-4.4.0-96-generic linux-headers-4.4.0-97 linux-headers-4.4.0-97-generic linux-headers-4.4.0-98 linux-headers-4.4.0-98-generic linux-image-4.4.0-101-generic
linux-image-4.4.0-103-generic linux-image-4.4.0-104-generic linux-image-4.4.0-108-generic linux-image-4.4.0-109-generic linux-image-4.4.0-112-generic linux-image-4.4.0-116-generic
linux-image-4.4.0-119-generic linux-image-4.4.0-121-generic linux-image-4.4.0-124-generic linux-image-4.4.0-127-generic linux-image-4.4.0-128-generic linux-image-4.4.0-130-generic
linux-image-4.4.0-133-generic linux-image-4.4.0-134-generic linux-image-4.4.0-137-generic linux-image-4.4.0-138-generic linux-image-4.4.0-139-generic linux-image-4.4.0-141-generic
linux-image-4.4.0-142-generic linux-image-4.4.0-143-generic linux-image-4.4.0-148-generic linux-image-4.4.0-150-generic linux-image-4.4.0-59-generic linux-image-4.4.0-66-generic
linux-image-4.4.0-70-generic linux-image-4.4.0-71-generic linux-image-4.4.0-72-generic linux-image-4.4.0-75-generic linux-image-4.4.0-78-generic linux-image-4.4.0-79-generic
linux-image-4.4.0-81-generic linux-image-4.4.0-83-generic linux-image-4.4.0-87-generic linux-image-4.4.0-89-generic linux-image-4.4.0-91-generic linux-image-4.4.0-92-generic
linux-image-4.4.0-93-generic linux-image-4.4.0-96-generic linux-image-4.4.0-97-generic linux-image-4.4.0-98-generic linux-modules-4.4.0-143-generic linux-modules-4.4.0-148-generic
linux-modules-4.4.0-150-generic
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
linux-headers-4.4.0-157
The following NEW packages will be installed:
linux-headers-4.4.0-157
0 upgraded, 1 newly installed, 0 to remove and 164 not upgraded.
8 not fully installed or removed.
Need to get 0 B/10.0 MB of archives.
After this operation, 71.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
(Reading database ... 1293295 files and directories currently installed.)
Preparing to unpack .../linux-headers-4.4.0-157_4.4.0-157.185_all.deb ...
Unpacking linux-headers-4.4.0-157 (4.4.0-157.185) ...
dpkg: error processing archive /var/cache/apt/archives/linux-headers-4.4.0-157_4.4.0-157.185_all.deb (--unpack):
unable to create '/usr/src/linux-headers-4.4.0-157/Kbuild.dpkg-new' (while processing './usr/src/linux-headers-4.4.0-157/Kbuild'): No space left on device
No apport report written because the **error message indicates a disk full error**
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/linux-headers-4.4.0-157_4.4.0-157.185_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
ubuntu#ip-172-21-0-17:/opt/aws-scripts-mon$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 395M 46M 350M 12% /run
/dev/xvda1 20G 13G 6.4G 67% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mapper/cassandra-volume 200G 6.4G 194G 4% /var/lib/cassandra
tmpfs 395M 0 395M 0% /run/user/1000
Your inode usage is high. Which basically means that you have a lot of small files on your server (since disk is still not full).
So you need to find these small files and delete old / not required ones.
Meanwhile, as a quick workaround, you may try autoremoving unneeded packages.
sudo apt-get autoremove
or
sudo apt autoremove
Note: This is not a permanent solution. It's just a quick fix.
You are bound to face this problem time and again, unless you find where these small files are getting created and write a cron job to periodically delete them.

EC2 no space left, can't find out what's using the space

I have a small ec2 instance (Amazon Linux release 2 (Karoo)) as a jenkins node. It just greeted me with "no space left". ssh into it and df -h shows:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 44M 3.9G 2% /dev/shm
tmpfs 3.9G 560K 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/xvda1 64G 63G 1.5G 98% /
tmpfs 798M 0 798M 0% /run/user/1000
As shown the usage of /dev/xvda1 is 98% (I just cleaned up tmp and cache).
But I can't find out what's consuming the space.
du -h --max-depth=1 | sort -hr from / shows:
...error messages of Permission denied
3.5G .
1.4G ./usr
1.2G ./home
588M ./var
200M ./opt
57M ./tmp
40M ./boot
18M ./etc
552K ./run
0 ./sys
0 ./srv
0 ./root
0 ./proc
0 ./mnt
0 ./media
0 ./local
0 ./dev
If I append sudo in front of it I get error like du: cannot access ‘./proc/24384/task/24384/fd/3’: No such file or directory
From what I've done, there seems nothing huge consuming my 64GB disk space.
What have I missed?
To find large files on the system we can use the 'find' command along with the -size parameter.
sudo find / -type f -size +1M -exec ls -lah {} \;
Change the +1M in the above example to meet your requirements. The example will find all files from the file system's root (and deeper), of type file, and larger than 1MB in size.
Example of output:
user#server:~$ sudo find / -type f -size +1M -exec ls -lah {} \;
-r--r--r-- 1 root root 3.4M Oct 17 2018 /root/.npm/_cacache/content-v2/sha1/a5/67/c2af000ad22dc8f2a653d91978ae1f5316f4
-r--r--r-- 1 root root 8.6M Jul 10 16:21 /lib/udev/hwdb.bin
-rw-r--r-- 1 root root 1.6M Jan 28 2018 /lib/udev/hwdb.d/20-OUI.hwdb
-rw-r--r-- 1 root root 1.2M Jan 28 2018 /lib/udev/hwdb.d/20-usb-vendor-model.hwdb
-rw-r--r-- 1 root root 2.8M Jan 28 2018 /lib/udev/hwdb.d/20-pci-vendor-model.hwdb
[... output suppressed ...]
It's of course very possible that the problem is a large number of small files taking up all the disk space. You'll have to analyse the output of the find command.
Found my problem. Long story short, is docker's overlay data accumulating.
more information
Because those filese are recognized as open files somehow, it's not showing as a result of du or ncdu. They can be viewed with lsof. Note they have "deleted" markers after them though.
docker system prune -f -a --filter "until=72h" reclaimed 50GB for me.

cannot find cgroup V2 controllor

guys
I've mounted cgroup V2 manually but I found there are none controllor in the root hierarchy.
[root#node5 /]# mount -t cgroup2 none /cgroup2/
[root#node5 /]# cd cgroup2/
[root#node5 cgroup2]# ls
cgroup.controllers cgroup.max.descendants cgroup.stat cgroup.threads
cgroup.max.depth cgroup.procs cgroup.subtree_control
[root#node5 cgroup2]# cat cgroup.controllers
[root#node5 cgroup2]# uname -r
5.0.0-rc6+
This is the expected behavior. It's really due to the fact that, very likely, you have both the older cgroups v1 as well as the newer cgroups v2 present, IOW, they're mixed. Now no controllers show up in v2.
What you can do is disable v1 and retry. To disable cgroups v1, reboot the box passing this kernel command-line parameter via your bootloader (GRUB, if on an x86):
cgroup_no_v1=all
After booting with this, pl mount and check, it should show up (provided of course it's configured in the kernel).

smlnj interactive system: fatal error

Getting a strange error on my Crouton chroot on my Chromebook after apt-get install smlnj and attempting to call sml from the terminal, an error that I haven't been able to find anywhere else:
$ sml
/usr/lib/smlnj/bin/sml: Error -- unable to map 1179648 bytes, errno = 1
/usr/lib/smlnj/bin/sml: Fatal error -- unable to allocate memory object for BIBOP
Any guidance?
Edit: This issue (bug #120) was fixed in SML/NJ version 110.77 (release notes) by enabling MAP_ANONYMOUS, which was previously unsupported in Linux. Installing the latest version of SMLNJ would be preferable to the solution below, since it doesn't require weakening the security of the system. If you do continue via modifying the permissions of /dev, it's advisable to remove exec permission after each session.
This problem is primarily reproducible in a Chromium OS environment running ChrUbuntu or Crouton, but may happen to other users in other environments who have a similar problem.
Running strace on sml gives this (attempted) operation before the error output:
open("/dev/zero", O_RDONLY) = 4
mmap2(NULL, 1245184, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 4, 0) = -1 EPERM (Operation not permitted)
According to the mmap(2) documentation, EPERM indicates that the system is attempting to allocate memory with execute permission in a directory that is marked noexec.
Indeed, mount gives:
...
devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=960456k,nr_inodes=240114,mode=755)
...
The solution is thus to remount /dev with exec permission:
$ sudo mount -o remount,exec /dev
$ sml
Standard ML of New Jersey v110.75 [built: Thu May 9 05:41:01 2013]
-