I am getting the message 'disk is full' despite having plenty of free space.
Inodes :-
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 1638400 55434 1582966 4% /
tmpfs 480574 1 480573 1% /dev/shm
/dev/xvdb2 3014656 3007131 7525 100% /export
Disk space :-
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 25G 16G 9.3G 63% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/xvdb2 46G 24G 20G 56% /export
Note: we have our magento application code which is under /dev/xvdb2 /export.please how to sort out the issue.
Related
Hint on centos7 when writing to file. No space left on device (28), but I checked the disk and Inodes both have space. I don't know why. Has this ever happened to anyone?
[root#GDI2390 sync_backup]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 52403200 2595000 49808200 5% /
devtmpfs 32867708 0 32867708 0% /dev
tmpfs 32874076 24 32874052 1% /dev/shm
tmpfs 32874076 279528 32594548 1% /run
tmpfs 32874076 0 32874076 0% /sys/fs/cgroup
/dev/sda1 508588 98124 410464 20% /boot
/dev/mapper/centos-home 4797426908 902326816 3895100092 19% /home
tmpfs 6574816 0 6574816 0% /run/user/0
[root#GDI2390 sync_backup]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/centos-root 52428800 89274 52339526 1% /
devtmpfs 8216927 562 8216365 1% /dev
tmpfs 8218519 2 8218517 1% /dev/shm
tmpfs 8218519 734 8217785 1% /run
tmpfs 8218519 13 8218506 1% /sys/fs/cgroup
/dev/sda1 512000 330 511670 1% /boot
/dev/mapper/centos-home 4797861888 26409024 4771452864 1% /home
tmpfs 8218519 1 8218518 1% /run/user/0
check "Disk space vs Inode usage"
df -h vs df -i
check also - No space left on device
Assuming this is ext[3|4], check and see if any of the folders you are rsyncing contains about 1M files. Note it is not a straight forward limit that has over flowed, it is more complicated and related to length of your filenames and depth of b-trees.
It is not disk space or inodes. It is the directory index that may have overflowed. See https://askubuntu.com/questions/644071/ubuntu-12-04-ext4-ext4-dx-add-entry2006-directory-index-full
I have an EC2 instance with 2 EBS volumes. On the root volume, I increased the volume size.
First I modified the volume size on console. Then followed this instruction to extended the partition following this instruction.
However, when running any command, I get No space left on device.
echo "Hello" > hello.txt
-bash: hello.txt: No space left on device
The df -h correctly shows, the total space correctly.
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 395M 596K 394M 1% /run
/dev/xvda1 30G 19G 9.6G 67% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/xvdg 10G 6.2G 3.9G 62% /data2
tmpfs 395M 0 395M 0% /run/user/1001
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 30G 0 disk
└─xvda1 202:1 0 30G 0 part /
xvdg 202:96 0 31G 0 disk /data2
The bottom of the guide said,
If the increased available space on your volume remains invisible to
the system, try re-initializing the volume as described in
Initializing Amazon EBS Volumes.
So I followed the instruction to reinitialize the volume.
I used dd to reinitialize.
sudo dd if=/dev/xvda of=/dev/null bs=1M
30720+0 records in
30720+0 records out
32212254720 bytes (32 GB, 30 GiB) copied, 504.621 s, 63.8 MB/s
However, I still get "No space left on device" error. I have done multipe reboot of the instance, I still see the same error.
Update 1: I have a large number of small files 4-10KB. So, I am running out of inodes. Please let me know how to increase the inodes (on ext4 partition)? Thanks.
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 497270 371 496899 1% /dev
tmpfs 504717 539 504178 1% /run
/dev/xvda1 1966080 1966080 0 100% /
tmpfs 504717 1 504716 1% /dev/shm
tmpfs 504717 4 504713 1% /run/lock
tmpfs 504717 18 504699 1% /sys/fs/cgroup
/dev/xvdg 5242880 39994 5202886 1% /data2
tmpfs 504717 10 504707 1% /run/user/1001
Please let me know how to resolve this error.
You don't need to reinitialise the volumes if you are using the new generations EC2 instance type like M4, M5, T2 and T3.
You have to also expand your volume on EC2 instance:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html
As I can see your Inode is full, try this command to find the directory:
for i in /*; do echo $i; find $i |wc -l; done
Then, after finding the directory, do:
for i in /path/to/directory/*; do echo $i; find $i |wc -l; done
After that remove those file using:
rm -rf /path/to/fileorfoldername
As you don't want to remove the files, you will have to create a new filesystem using mke2fs and -I parameter for inode size, or keep increasing your root volume more.
Here is a similar question for it:
https://unix.stackexchange.com/questions/26598/how-can-i-increase-the-number-of-inodes-in-an-ext4-filesystem
Also, you can move your files to secondary drive.
You are running out of inodes on your / (root) directory. so to debug it, you need:
Find what are the inodes pointing to or open files something like for i in /*; do echo $i; find $i |wc -l; done
Now look for directories which take more time and then delete that dir
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 4 days ago.
Improve this question
l try to install pytorch on amazon as follow :
conda install pytorch torchvision cuda80 -c soumith
l get the following error :
Package plan for installation in environment /home/ubuntu/anaconda2/envs/crnn:
The following NEW packages will be INSTALLED:
cuda80: 1.0-0 soumith
pytorch: 0.1.12-py27_2cu80 soumith [cuda80]
torchvision: 0.1.8-py27_2 soumith
Proceed ([y]/n)? y
CondaError: IOError(28, 'No space left on device')
CondaError: IOError(28, 'No space left on device')
CondaError: IOError(28, 'No space left on device'
when l run the following command :
df -h
l get the following :
Filesystem Size Used Avail Use% Mounted on
udev 30G 0 30G 0% /dev
tmpfs 6.0G 8.6M 6.0G 1% /run
/dev/xvda1 7.7G 7.4G 361M 96% /
tmpfs 30G 0 30G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 30G 0 30G 0% /sys/fs/cgroup
tmpfs 6.0G 0 6.0G 0% /run/user/1000
now l added a new partition called /data which has 50 Go :
/data$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 30G 0 30G 0% /dev
tmpfs 6.0G 8.9M 6.0G 1% /run
/dev/xvda1 7.7G 6.3G 1.4G 82% /
tmpfs 30G 0 30G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 30G 0 30G 0% /sys/fs/cgroup
/dev/xvdf1 50G 3.9G 43G 9% /data
tmpfs 6.0G 0 6.0G 0% /run/user/1000
But most of installation file system such as cuda have to be installed in /
How can l transfer the 50 G0 free space from /data to / so that
/dev/xvda1 will have 58 GO ?
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
`-xvda1 202:1 0 8G 0 part /
xvdf 202:80 0 50G 0 disk
`-xvdf1 202:81 0 50G 0 part /data
Thank you a lot
I have cPanel & WHm installed on an EC2 instance.
The EC2 EBS volume is 100GB
CentOS 6 x86_64 (2014_09_29) EBS pv-1-adc4348e-1dc3-41df-b833-e86ba57a33d6-ami-809020e8.2 (ami-bc8131d4)
This is what df -h shows:
Filesystem Size Used Avail Use% Mounted on
/dev/xvde1 7.8G 6.6G 834M 89% /
tmpfs 3.7G 0 3.7G 0% /dev/shm
/usr/tmpDSK 485M 11M 449M 3% /tmp
The EBS is mounted at /dev/sda
I am trying to increase the size of the xvde1 to almost fill the 100GB
When I run resize2fs /dev/xvde1 i get this:
root#vs1 [~]# resize2fs /dev/xvde1
resize2fs 1.41.12 (17-May-2010)
The filesystem is already 2096896 blocks long. Nothing to do!
I have tried the solution stated here: EC2 Can't resize volume after increasing size
But when i reboot the instance it gets stuck on 1/2 Status Checks
I saw something that stated to start the partition on 16 and also to use the same start as it is in the original configuration. This did not work.
Here is some more information and other things that i have tried:
root#vs1 [~]# resize2fs /dev/xvde1
resize2fs 1.41.12 (17-May-2010)
The filesystem is already 2096896 blocks long. Nothing to do!
root#vs1 [~]# resize2fs /dev/xvde
resize2fs 1.41.12 (17-May-2010)
resize2fs: Device or resource busy while trying to open /dev/xvde
Couldn't find valid filesystem superblock.
root#vs1 [~]# resize2fs /dev/xvde1
resize2fs 1.41.12 (17-May-2010)
The filesystem is already 2096896 blocks long. Nothing to do!
root#vs1 [~]# resize2fs /dev/xvde1 +25G
resize2fs 1.41.12 (17-May-2010)
The containing partition (or device) is only 2096896 (4k) blocks.
You requested a new size of 6553600 blocks.
root#vs1 [~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvde1 7.8G 6.6G 834M 89% /
tmpfs 3.7G 0 3.7G 0% /dev/shm
/usr/tmpDSK 485M 11M 449M 3% /tmp
root#vs1 [~]# resize2fs /dev/xvde1 +15g
resize2fs 1.41.12 (17-May-2010)
The containing partition (or device) is only 2096896 (4k) blocks.
You requested a new size of 3932160 blocks.
root#vs1 [~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvde1 7.8G 6.6G 834M 89% /
tmpfs 3.7G 0 3.7G 0% /dev/shm
/usr/tmpDSK 485M 11M 449M 3% /tmp
root#vs1 [~]# resize2fs /dev/xvde1
resize2fs 1.41.12 (17-May-2010)
The filesystem is already 2096896 blocks long. Nothing to do!
root#vs1 [~]# fdisk -l
Disk /dev/xvde: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00098461
Device Boot Start End Blocks Id System
/dev/xvde1 * 1 1045 8387584 83 Linux
root#vs1 [~]# Write failed: Broken pipe
This is the last step in a complete build that has taken over 18 hours.
I may just be delirious. Any and all help will be greatly appreciated!
EDIT:
The system log on the server shows this, not sure if it is any help:
Thread "main": pointer: 0x21e0003760, stack: 0x3740000
"main" "root=/dev/sda" "ro" "4"
vbd 2048 is hd0
******************* BLKFRONT for device/vbd/2048 **********
backend at /local/domain/0/backend/vbd/60/2048
209715200 sectors of 512 bytes
**************************
[H[J
GNU GRUB version 0.97 (7864320K lower / 0K upper memory)
[m[4;2H+-------------------------------------------------------------------------+[5;2H|[5;76H|[6;2H|[6;76H|[7;2H|[7;76H|[8;2H|[8;76H|[9;2H|[9;76H|[10;2H|[10;76H|[11;2H|[11;76H|[12;2H|[12;76H|[13;2H|[13;76H|[14;2H|[14;76H|[15;2H|[15;76H|[16;2H|[16;76H|[17;2H+-------------------------------------------------------------------------+[m
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.[5;78H [m[7m[5;3H CentOS (2.6.32-504.el6.x86_64) [5;75H[m[m[6;3H CentOS (2.6.32-431.29.2.el6.x86_64) [6;75H[m[m[7;3H [7;75H[m[m[8;3H [8;75H[m[m[9;3H [9;75H[m[m[10;3H [10;75H[m[m[11;3H [11;75H[m[m[12;3H [12;75H[m[m[13;3H [13;75H[m[m[14;3H [14;75H[m[m[15;3H [15;75H[m[m[16;3H [16;75H[m[16;78H [5;75H[23;4H The highlighted entry will be booted automatically in 1 seconds. [5;75H[H[J Booting 'CentOS (2.6.32-504.el6.x86_64)'
root (hd0,0)
Filesystem type is ext2fs, partition type 0x83
kernel /boot/vmlinuz-2.6.32-504.el6.x86_64 ro root=LABEL=_ console=ttyS0,115200
crashkernel=no SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=us
initrd /boot/initramfs-2.6.32-504.el6.x86_64.img
============= Init TPM Front ================
Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT
Tpmfront:Info Shutting down tpmfront
Before resizing the filesystem by "resize2fs" command you should first resize your partition:
let's list block devices attached to our box:
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 16G 0 disk
└─xvda1 202:1 0 8G 0 part /
As you can see /dev/xvda1 is only 8 GiB partition on a 16 GiB device and there are no other partitions on the volume.
step-1) We will use "growpart" to resize 8G partition up to 16G:
# install "cloud-guest-utils" if it is not installed already
apt install cloud-guest-utils
# resize partition
growpart /dev/xvda 1
Let's check the result (you can see /dev/xvda1 is now 16G):
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 16G 0 disk
└─xvda1 202:1 0 16G 0 part /
Lots of SO answers suggest to use fdisk with delete / recreate partitions, which is nasty, risky, error-prone process especially when we change boot drive.
step-2) resize file system to grow all the way to fully use new partition space
# Check before resizing ("Avail" shows 1.1G):
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 6.3G 1.1G 86% /
# resize filesystem
resize2fs /dev/xvda1
# Check after resizing ("Avail" now shows 8.7G!-):
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 16G 6.3G 8.7G 42% /
And by the way, there's no need to stop instance and detach EBS volume to resize it anymore! 13-Feb-2017 Amazon announced: "Amazon EBS Update – New Elastic Volumes Change Everything" See my other SO answer for details.
Update: Use sudo xfs_growfs /dev/xvda1 instead of resize2fs when XFS filesystem
This answer is dangerous, hasn't been verified since 2016, and has the potential to delete your partition if you're not aware of what's happening. Use with caution, and please reference #Dmitry's answer below as well.
To expand on JD's answer, here's exactly what to do:
df -h #print the name of your boot partition
lsblk #show info on all your block devices
You'll see from that output what the name of the disk is of your root partition. For example, you probably see something like this:
xvde 202:64 0 32G 0 disk
└─xvde1 202:65 0 8G 0 part /
Our goal is to make xvde1 use the whole available space from xvde.
Here's how to resize your partition:
fdisk /dev/xvda (the disk name, not your partition)
This enters into the fdisk utility.
u #Change the display to sectors
p #Print info
d #Delete the partition
n #New partition
p #Primary partition
1 #Partition number
2048 #First sector
Press Enter to accept the default
p #Print info
a #Toggle the bootable flag
1 #Select partition 1
w #Write table to disk and exit
Now, reboot your instance:
reboot
After it comes back do:
resize2fs /dev/xvde1 (the name of your partition, not the block device)
And finally verify the new disk size:
df -h
After searching and searching with no answer here, i FINALLY came across the answer!
"fdisk, put it into 'units' mode by typing 'u' then 'p' to print the partition table as it is now and write down the starting sector of the existing partition. Then delete that partition and make a new one, with the same partition number and starting on exactly the same sector and ending at the end of the disk, make sure that partition is 'active' then save your changes and reboot. Once you reboot after that, you should be able to resize2fs the existing filesystem to take up all space. Backups are recommended and it's your data to lose!!"
Putting it into sectors via Units mode helped me select the proper start and end of the FS!!
Credit: https://www.centos.org/forums/viewtopic.php?t=4783
I know the question is answered already. But still, if someone is stuck with the problem and need a more detailed step by step explanation. You can check out the official Amazon docs. This has the steps to increase the size and specific commands if you are using the "ec2-user (amazon ami)" instance.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html
also if you are using windows instance. You can visit
"For information about extending a Windows file system, see Extending a Windows file system after resizing a volume in the Amazon EC2 User Guide for Windows Instances."
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/recognize-expanded-volume-windows.html
None of the above answers helped me. Here is something to consider for centos.
I wanted to extend my root volume from 8GB to 16GB
[root#ip-10-0-0-218 ~]# cat /etc/centos-release
CentOS Linux release 8.2.2004 (Core)
Determine the current state
[root#ip-10-0-0-218 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 60G 0 disk
├─nvme0n1p1 259:1 0 600M 0 part /boot/efi
└─nvme0n1p2 259:2 0 7.8G 0 part /
First attempt to increase space (failed)
[root#ip-10-0-0-218 /]# xfs_growfs -d /dev/nvme0n1p2
meta-data=/dev/nvme0n1p2 isize=512 agcount=4, agsize=508800 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=2035200, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data size unchanged, skipping
Use Parted
[root#ip-10-0-0-218 ~]# parted
GNU Parted 3.2
Using /dev/nvme0n1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) resizepart 2 +20GiB
Warning: Partition /dev/nvme0n1p2 is being used. Are you sure you want to continue?
Yes/No? Yes
End? [8966MB]? 16966MB
quit
Now extend the size (success)
[root#ip-10-0-0-218 ~]# xfs_growfs /dev/nvme0n1p2
meta-data=/dev/nvme0n1p2 isize=512 agcount=4, agsize=508800 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=2035200, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 2035200 to 3988233
Verify
[root#ip-10-0-0-218 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.7G 0 1.7G 0% /dev
tmpfs 1.7G 0 1.7G 0% /dev/shm
tmpfs 1.7G 22M 1.7G 2% /run
tmpfs 1.7G 0 1.7G 0% /sys/fs/cgroup
/dev/nvme0n1p2 16G 7.1G 8.2G 47% /
/dev/nvme0n1p1 599M 6.4M 593M 2% /boot/efi
tmpfs 345M 0 345M 0% /run/user/1000
1. Confirm the kernel.
~]# uname -r
2.6.32-431.29.2.el6.x86_64
2. Confirm the initial allocated size
]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 797M 6.7G 11% /
tmpfs 498M 0 498M 0% /dev/shm
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 **15G 0 disk**
└─xvda1 202:1 0 **8G 0 part /**
3. Install EPEL6 Repo
~]# wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
~]# rpm -ivh epel-release-6-8.noarch.rpm
3. Install the dracut & cloud-utils growpart modules-growroot
~]# yum install -y dracut-modules-growroot cloud-utils-growpart
4. The initrd needs to be rebuilt after installing the dracut-modules-growroot.
Create a backup copy of the current initramfs:
~]# cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
(ii)Create the initramfs for the current kernel:
~]# dracut -f
4. Result after reboot
~]# reboot
~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 **15G 0 disk**
└─xvda1 202:1 0 **15G 0 part /**
~]# resize2fs /dev/xvda1
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/xvda1 is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/xvda1 to 3931652 (4k) blocks.
The filesystem on /dev/xvda1 is now 3931652 blocks long.
~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 **15G** 932M 14G 7% /
tmpfs 498M 0 498M 0% /dev/shm
I need to make a ramfs an mount it to an directory in linux using c++. I want to make it like a user (no sudo).
I need to call an application on a file that i created and it will be often. Writing it to HDD is very slow.
I found just:
system("mkdir /mnt/ram");
system("mount -t ramfs -o size=20m ramfs /mnt/ram");
but that is not good. I want to be a regular user, and command mount can be called just as root.
what can i do?
For a userspace ramfs solution, you can use python-fuse-ramfs.
I checket if /tmp is a ramfs, but it is not. It creates files on the HDD. but when i run df -h it outputs:
rootfs 25G 9,4G 15G 40% /
devtmpfs 1,9G 0 1,9G 0% /dev
tmpfs 1,9G 1,6G 347M 83% /dev/shm
tmpfs 1,9G 1,3M 1,9G 1% /run
/dev/mapper/vg_micro-root 25G 9,4G 15G 40% /
tmpfs 1,9G 0 1,9G 0% /sys/fs/cgroup
tmpfs 1,9G 0 1,9G 0% /media
/dev/mapper/vg_micro-stack 289G 191M 274G 1% /stack
/dev/mapper/vg_micro-home 322G 40G 266G 14% /home
/dev/sda2 485M 89M 371M 20% /boot
/dev/sda1 200M 19M 182M 10% /boot/efi
This means that tmpfs (ramdisks) are: /dev/shm, /run, /sys/fs/cgroup and /media. But only one of this is meant to be a temporary ramdisk for comunication between processes, using files. Here is the /dev/shm description and usage. The only thing is that tmpfs will not grow dynamically, but for my purposes it will be enough (20MB - 1GB).