I am trying to run a sentiment analysis on Google Cloud Platform (AI Platform). When I try to split the data into training, Its showing memory error like below error like below
MemoryError: Unable to allocate 194. GiB for an array with shape (414298,) and data type <U125872
How do I increase the memory size accordingly? Should I change the machine type in the instance? If so Which setting would be appropriate?
From the error it's seems the VM is out of memory.
1 - Create a new Notebook with another Machine type.
For this, go to AI Platform > Notebooks and click on NEW INSTANCE. Select the option most fit you (R 3.6, Python 2 and 3, etc.) and click on ADVANCED OPTIONS in the popped pane. In the Machine Configuration area you can pick a Machine type with more memory.
Please start with n1-standard-16 or n1-highmem-8, and if any of those works, jump to n1-standard-32 or n1-highmem-16.
Using the command you also can change the machine size:
gcloud compute instances set-machine-type INSTANCE_NAME \
--machine-type NEW_MACHINE_TYPE
2 - Change the dtype.
If you are working with np.float64 type, you can change it to np.float32 in order to reduce size. So you can modify the line:
result = np.empty(self.shape, dtype=dtype)
By:
result = np.empty(self.shape, dtype=np.float32)
If you don't want to modify your code I suggest you to follow first option.
Changing machine type to one with enough resources is necessary but might not be sufficient. As indicated here Jupyter as a service settings need to be set to allow for greater memory usage. Make sure of this trying the following steps:
Open a terminal on your Jupyter instance and run the following command:
sudo nano /lib/systemd/system/jupyter.service
Check if the MemoryHigh and MemoryMax parameters on the text editor that prompts (like the one showed bellow) are set to your desired capacity. If not, then change them.
[Unit]
Description=Jupyter Notebook
[Service]
Type=simple
PIDFile=/run/jupyter.pid
CPUQuota=97%
MemoryHigh=3533868160
MemoryMax=3583868160
ExecStart=/bin/bash --login -c '/opt/conda/bin/jupyter lab --config=/home/jupyter/.jupyter/jupyter_notebook_config.py'
User=jupyter
Group=jupyter
WorkingDirectory=/home/jupyter
Restart=always
[Install]
WantedBy=multi-user.target`
Save and exit.
Finally, run the following command on the terminal:
echo 1 | sudo tee /proc/sys/vm/overcommit_memory
This will allow for full usage of the vm resources on the Jupyter instance.
I want, when my centos 7 server boots, to run
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
because I want to reuse open connections.
Systemd is allready installed, but my command is not a service, just a one time execution command at startup.
How can I run automatically this command at startup ? Thanks !
Add your command to
/etc/rc.d/rc.local
and it will run at startup.
The correct way to make the changes persistent is to edit the file:
/etc/sysctl.conf
After the change, type:
sysctl -p
This will load the changes into the current session. The fact that the settings are in /etc/sysctl.conf will ensure that they load on reboots.
I tried using VBoxManage guestproperty wait <vmname> ... but what looked like obvious patterns didn't work. I'm writing a script which imports a new VM, configures it, launches it, takes a snapshot, and then closes it, and obviously I need to know when the VM is running before taking the final two steps.
Thanks.
You can use showvminfo output:
for Linux:
VBoxManage showvminfo "vm_name" | grep State
for Windows:
VBoxManage showvminfo "vm_name" | findstr State
See the thread below:
https://unix.stackexchange.com/questions/28611/how-to-automatically-start-and-shut-down-virtualbox-machines
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
I've run out of space on a virtual machine disk which is a vmdk and need to resize the virtual image. Resizing with the command
vboxmanage modifyhd Machine-disk1.vmdk --resize 30720
gives the error
Progress state: VBOX_E_NOT_SUPPORTED
VBoxManage: error: Resize hard disk operation for this format is not implemented yet!
How can I resize a vmdk image?
If you want to end having back a vmdk hard disk (maybe you are interested in using the disk in vwmare too), the complete steps are:
VBoxManage clonemedium "source.vmdk" "cloned.vdi" --format vdi
VBoxManage modifymedium "cloned.vdi" --resize 51200
VBoxManage clonemedium "cloned.vdi" "resized.vmdk" --format vmdk
The above will resize the hard disk up to 50GB (50 * 1024MB).
To complete things you need to resize the drive too! To achieve this, you might want to download gparted iso and boot from that iso to resize your drive (select the iso from within the virtualbox settings).
P.S. If your new size is too small, you'll get the same error even with your new vdi file.
I was able to resize the vmdk by cloning it and then modifying and resizing.
vboxmanage clonehd "virtualdisk.vmdk" "new-virtualdisk.vdi" --format vdi
vboxmanage modifyhd "new-virtualdisk.vdi" --resize 30720
I have a Windows 7 client on a Mac host and this post was VERY helpful. Thanks.
I would add that I didn't use gparted. I did this:
Launch new enlarged vmdk image.
Go to Start and right click Computer and select Manage.
Click Disk Management
You should see some grayed space on your (in my case) C drive
Right click the C drive and select Extend Volume.
Choose size and go
Sweet! I preferred that to using a 3rd party tool with warnings about data loss.
Cheers!
vmdk's :
Rather fixed size allocation (step 1,2).
Even after expansion, not readily available inside the vmdk's OS (step 3,4,5)
STEPS:
1) convert to ".vdi" first - VBoxManage clonehd v1.vmdk v1.vdi --format vdi
2) expand the size using command-line (Ref: tvial's blog for step by step info)
OR
expand from the Virtual Media Manager in VirtualBox.
[ NOW - INSIDE VM ]
3) Expand the size of drive, with new allocation (e.g. for Ubuntu running on virtual-machine : use GParted)
4) Extend the filesystem - lvextend -L +50G <file-system-identifier>
ILLUSTRATION:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
...
sda 8:0 0 200G 0 disk
└─sda1 8:1 0 200G 0 part
├─myfs-vg-cloud 253:0 0 99G 0 lvm /
└─myfs-vg-swap-1 253:1 0 980M 0 lvm [SWAP]
$ lvextend -L +100G /dev/mapper/myfs-vg-cloud
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
...
sda 8:0 0 200G 0 disk
└─sda1 8:1 0 200G 0 part
├─myfs-vg-cloud 253:0 0 199G 0 lvm /
└─myfs-vg-swap-1 253:1 0 980M 0 lvm [SWAP]
5) Extend the "/home" - resize2fs <file-system-identifier>
ILLUSTRATION:
$ df -h /home/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/myfs-vg-cloud 97G 87G 6.0G 94% /
$ resize2fs /dev/mapper/myfs-vg-cloud
$ df -h /home/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/myfs-vg-cloud 196G 87G 101G 47% /
Your system must now be ready to use, with extended allocations !!
Actually, Only this these commands are needed:
VBoxManage clonehd "source.vmdk" "cloned.vdi" --format vdi
VBoxManage modifyhd "cloned.vdi" --resize 51200
Then, you can select cloned.dvi in Virtualbox GUI storage.
After that, start the virtual windows and expand your C disk as the methods of Code Chops.
It is not necessary to convert the *.vdi file to *.vmdk file back.
Since this is a vmdk file, you could use VMWare's vdiskmanager, if it's available for your platform. VMWare has x86 Linux, Windows, and OS X versions here (see "Attachments" on the right rail).
And then you just do:
1023856-vdiskmanager-windows-7.0.1.exe -x 30720M Machine-disk1.vmdk
It avoids having to clone, then expand the disk. Now, the downside is you need the extra tool, and vmdk is VMWare's disk format, and you're still using Virtualbox, so there could be incompatibilities.
qemu-img might also work, but I'm not sure if it supports resizing vmdk files. It would look something like:
qemu-img resize Machine-disk1.vmdk +8G
And just a reminder, with both, you still have to grow the partition after resizing the underlying disk. All these tools are essentially dd if=/dev/old_disk of=/dev/new_disk bs=16M.
Tried all the solutions here, but nothing seemed to work. After hours of digging I found this blog post and like magic everything works. I had to make little adjustments so here is the modified version. Tested with Docker version 17.06.0-ce, build 02c1d87.
Once developers really start containerising their applications, they often generate a large number of images and quickly fill up the 20GB hard drive space allocated to the Docker virtual machine by default. To make sure the Docker virtual machine has plenty of disk space, we should resize /dev/sda1 to a number that is more reasonable.
Download GParted Live CD/USB/HD/PXE Bootable Image.
Stop the Docker virtual machine docker-machine stop default.
Boot2Docker package installer ships with a VMDK volume, which VirtualBox’s native tools cannot resize. In order to resize the Docker disk volume, first clone the VDI volume from the default VMDK volume vboxmanage clonehd /full/path/to/disk.vmdk /full/path/to/disk_resized.vdi --format VDI --variant Standard.
Resize the newly cloned VDI volume to the desired capacity. We recommend at least 64GB vboxmanage modifyhd /full/path/to/disk_resized.vdi --resize <size in MB>.
Launch the VirtualBox application, select default VM and click on the “Settings” gear on top.
Click on the “Storage” icon. Remove the default VMDK volume.
Add a new IDE controller.
Mount the GParted ISO via the “Add CD/DVD Device” option.
Mount the cloned VDI volume via the “Add CD/DVD Device” option.
If you are running Docker on a computer with a solid-state hard drive, please make sure the “Solid-state Drive” option is selected for the cloned VDI volume.
Click on the “Start” icon to boot up the Docker virtual machine, which will launch the GParted ISO. Select “GParted Live (Default settings)”.
Set the policy for handling keymaps to “Don’t touch keymap”.
Set language preference to option “33”, which maps to “US English”.
Select option “1” to run “Forcevideo” and configure X manually.
Keep the default resolution of “1024×760” by selecting option “2”.
Keep the default “vesa” as the VGA card.
Keep the default colour depth of “24” by selecting option “0”.
Once GParted launches, click on the “Resize/Move” icon.
Set the new disk volume size to desired size by dragging the slider. In this example, the maximum size is 127,035MB. Click on the “Resize/Move” button to start the process.
Confirm the resizing operation by clicking on the “Apply” button.
Power off the machine after the resizing operation finishes.
Remove the GParted ISO.
Log into the Docker virtual machine to verify that the volume resizing was successful by starting machine docker-machine start default
If you experience network issues, restart the docker-machine
You can use Vmware player to resize a vmdk.
This removes the round trip conversion (noted in other posts) if you want to keep it in vmdk format. Just make sure that when you add the vmdk into vmware, don't click upgrade format when prompted, keep the current one in case VirtualBox doesn't accept it.
For shrinking image size I use the following steps:
On the client machine cat /dev/zero > z;sync;sleep 3;sync;rm -f z
Shut down client machine
On the host VBoxManage clonehd "source.vmdk" "cloned.vdi" --format vdi
On the host VBoxManage modifyhd cloned.vdi --compact
On the host VBoxManage clonehd "cloned.vdi" "resized.vmdk" --format vmdk
As mentioned in the answer by Luca the required steps are:
Clone the VMDK image to VDI format
VBoxManage clonehd "source.vmdk" "cloned.vdi" --format vdi
Resize the VDI image
VBoxManage modifyhd "cloned.vdi" --resize 51200
Clone back to VMDK format
VBoxManage clonehd "cloned.vdi" "resized.vmdk" --format vmdk
Step 3 is optional if you can use a VDI image instead.
All these steps can be carried out in the VirtualBox graphical user interface as well, under File -> Virtual Media Manager. Select the drive you want to clone, and then choose Copy. A dialog opens that allows you to choose the disk format and options. To resize a VDI image, just use the slider in the bottom (click on Properties if it's not visible).
Finally, you may need to change the disk size in the guest OS as well:
Run the guest OS and resize the disk partition(s). If you are running Linux, you can use gparted or kparted.
VirtualBox for Windows
Resizing your disk file while preserving your virtual machine settings!
Step 1 - Resize the disk file
Start cmd.exe
cd to Oracle VM VirtualBox's dir (on 64-bit systems: "C:\Program Files\Oracle\VirtualBox\")
Run these commands (as above):
VBoxManage clonehd "C:\path\to\source.vmdk" "C:\path_to\cloned.vdi" --format vdi
VBoxManage modifyhd "C:\path\to\cloned.vdi" --resize 51200
Windows explorer and "copy address as text" via the address bar should help you get the path you need.
On windows system, The VirtaulBox VM directory underneath your user may contain an XML formatted database file of settings you've configured for your VM. Rename this file, with a .bak extension (it has a .vbox extension). Rename the original .vmdk file with a .bak extension as well to avoid another error. You can now safetly perform the third step without the error message to convert the machine back to .vmdk format, or the "duplicate disk" error.
VBoxManage clonehd "C:\path_to\cloned.vdi" "C:\path_to\source.vmdk" --format vmdk
You will be presented with a UID token. Copy this token by drag-highlighting it from the Windows Command Interpetor window and using the Ctrl+C keyboard shortcut.
Open the .vbox.bak file in a text editor such as Notepad++. You'll be presented with an XML-like database file. Look for these lines:
<VirtualBox xmlns="http://www.virtualbox.org/" version="1.16-windows">
<Machine uuid="{some uid}" name="source disk name" OSType="the_vbox_OS" snapshotFolder="Snapshots" lastStateChange="2043-03-23T00:54:18Z">
<MediaRegistry>
<HardDisks>
<HardDisk uuid="{some uid}" location="C:\path_to\source.vmdk" ...
On the line <HardDisk uuid="{some uid}" location="C:\path_to\source.vmdk" ..., delete the old UID token between the brackets and paste the one you copied from the command window. Make sure that you leave the brackets in place!
Save this file, and exit your text editor. Rename the .vbox.bak file to give it back its expected extension of .vbox.
Step 2 - Remove the junk
It is now safe to remove the .bak files remaining in the directory. What remains is a resized .vmdk with an updated .vbox database while with your previously preserved VirtualBox Manager settings.
Step 3 - Resize the disk's partition to fill the free space
You can now start the VirtualBox VM Manager and execute your VM, using the appropriate tools for the operating system to fill the new free space.
For Windows VMs, use diskpart from the command prompt booted from the Windows Recovery Consule (recovery partition) to SELECT DISK 1, LIST PARTITION and gather the partition number of your C:\ drive, then SELECT PARTITION #. You can use the EXTEND SIZE=mb to resize the Windows C:\ drive to the appropriate value. Make sure you leave room for the recovery and boot partitions! It's safe to subtract 4096 MB from your new virtual disk size to get this value, because of shadow copy and windows recovery files.
For Linux VMs, a live .ISO of gparted you can boot with the VM's disk file can be found at: http://gparted.org/ It will get you straight into a graphical user interface-based gparted-gtk, from where you can fill your free space.
For PPC / Mac VMs, Disk Utility from the Finder will asisst you in filling the free space, but you may want to consider the gparted Linux option, as currently the only method of which to boot MacOSX in VirtualBox is hackintosh, and you cannot extend your volume while booted into MacOSX. You may also want to seek out tweaking the VM's settings temporarily for gparted, to get it to boot. MacOSX partitions are recognized by gparted as HFS - "Heaping File System" partitions.
Step 4 - Cat Photos
Because the internet. ;) You're finished. Enjoy your new resized virtual .vmdk disk image with VirtualBox for Windows!
Here's a way to resize your VirtualBox disk, regardless of whether it is a fixed format or dynamic format disk. Specifically, it prevents the error you had when you disk is fixed-format.
⚠️ Backup the virtual disk. You never know what might go wrong.
On your host:
Open a terminal window.
On Windows: Open the command prompt cmd.
Go to the directory with the virtual disk you want to resize. For example:
cd "My VMs"
Create a new VirtualBox disk with your desired filename, size (in megabytes) and format (either Standard (dynamic) or Fixed). For example, to create a 50 GB fixed-format disk called MyNewDisk.vdi:
VBoxManage createmedium --filename "MyNewDisk.vdi" --size 50000 --variant Fixed
If VBoxManage is not recognized as a command, specify the full path to it. It can be found in the VirtualBox installation directory. On Windows the above command would become:
"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" createmedium
--filename "MyNewDisk.vdi" --size 50000 --variant Fixed
Copy the original disk to the new disk.
VBoxManage clonemedium "MyOriginalDisk.vdi" "MyNewDisk.vdi" --existing
The resize is done! You can check the properties of the new disk if you want:
VBoxManage showmediuminfo "MyNewDisk.vdi"
Change the virtual machine to use the new disk instead.
Next, on your guest OS you need to resize the partitions to use the newly available space.
I've got here because I needed to resize a disk for my Docker (CoreOS) development environment.
CoreOS docs says there's no need to resize the OS partition - that's bogus. After you resize the virtual disk you should follow these instructions and resize the OS partition via GParted:
https://docs.docker.com/articles/b2d_volume_resize/
A quick and simple option:
Turn off machine
Create new virtual box disk with desired size
Move content from old disk to inside new disk:
vboxmanage clonehd "source-disk.vmdk" "new-disk.vmdk" --existing
Turn on machine
You may have to resize partition in OS to fill the rest of the disk
Warning
If new disk is shorter than source, you will loss the data located from the new disk size position on the source disk.
You may prevent this deallocating delta space, where delta = size(source-disk) - size(new-disk), at the end of source disk inside OS before step 1.
Use these simple steps to resize the vmdk.
Click the File -> Virtual Media Player
Select vdi file and click properties
Here you can increase or decrease the vdi size.
I'm trying to change TCP/UDP of a virtual machine using VBoxManage setextradata.
Whenever I type the command:
sudo VBoxManage setextradata Windows_7 "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestEmule_TCP/Protocol" TCP
I get the following error:
VirtualBox Command Line Management Interface Version 2.1.4
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved.
[!] FAILED calling a->virtualBox->FindMachine(Bstr(a->argv[0]), machine.asOutParam()) at line 3688!
[!] Primary RC = VBOX_E_OBJECT_NOT_FOUND (0x80BB0001) - Object corresponding to the supplied arguments does not exist
[!] Full error info present: true , basic error info present: true
[!] Result Code = VBOX_E_OBJECT_NOT_FOUND (0x80BB0001) - Object corresponding to the supplied arguments does not exist
[!] Text = Could not find a registered machine named 'Windows_7'
[!] Component = VirtualBox, Interface: IVirtualBox, {339abca2-f47a-4302-87f5-7bc324e6bbde}
[!] Callee = IVirtualBox, {339abca2-f47a-4302-87f5-7bc324e6bbde}
The virtual-machine has been created using GUI. Any idea?
$ cd /Users/marco/Library/VirtualBox/Machines/Windows_7
$ ls
Logs Windows_7.xml
Windows 7.xml.1.5-macosx.bak
$ sudo vboxmanage registervm Windows_7.xml
VirtualBox Command Line Management Interface Version 2.1.4
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved.
[!] FAILED calling a->virtualBox->OpenMachine(Bstr(a->argv[0]), machine.asOutParam()) at line 762!
[!] Primary RC = NS_ERROR_FAILURE (0x80004005) - Operation failed
[!] Full error info present: true , basic error info present: true
[!] Result Code = NS_ERROR_FAILURE (0x80004005) - Operation failed
[!] Text = Could not lock the settings file '/var/root/Library/VirtualBox/Windows_7.xml' (VERR_FILE_NOT_FOUND)
[!] Component = Machine, Interface: IMachine, {ea6fb7ea-1993-4642-b113-f29eb39e0df0}
[!] Callee = IVirtualBox, {339abca2-f47a-4302-87f5-7bc324e6bbde}
It fails because you are using sudo. VirtualBox is designed to be run by any user (in the vboxusers group), and sudo runs the command as the root user whose VirtualBox configuration is empty.
You can check that by typing:
sudo VBoxManage -nologo list vms # Should print only a newline
VBoxManage -nologo list vms # Detailled information about all your VMs
!! WINDOWS ONLY!!
If you are not on an admin account and are trying to modify your VM in a administrator cmd window, type these commands:
cd "C:\Program Files\Oracle\VirtualBox"
VboxManage registervm "C:\Users\Your Name Here\VirtualBox VMs\Your VM name here\Your VM name here.vbox"
Now run your virtual box modify commands or what ever else you are doing and it should work!
Not a direct answer, but just to put it out there for other people searching for it:
On Mac OS X, you can tell VirtualBox to load VMs from another user's home directory, provided the file permissions allow it, or if you are running VirtualBox as the root user using sudo (e.g. if you absolutely have to access your host's web server on port 80).
The way to do this is to set VBOX_USER_HOME appropriately, e.g.
VBOX_USER_HOME=/Users/the_other_user/Library/VirtualBox
If you want to run VBoxHeadless under root, use:
sudo VBOX_USER_HOME=/Users/your_user_id/Library/VirtualBox nohup \
VBoxHeadless -s "IE10 - Win7" </dev/null &>/dev/null &
I had a similar error message, whenever I used sudo to start VBoxSDL:
Error: machine with the given name not found!
Check if this VM has been corrupted and is now inaccessible.
And similar to ypocat's answer, I solved it for Ubuntu using a small script like this:
#!/bin/bash
export VBOX_USER_HOME=/home/username/.config/VirtualBox
VBoxSDL --startvm nameOfVM
You can use it whenever you need to start your VM as root.
SOLUTION_1:
Missing Virtual Technology might be the reason. For Intel systems they have Intel VT-x (for AMD they have AMD-V), so make sure it is enabled. You can enable it in the boot screen go to BIOS Setup in that look for System Configuration tab and enable Virtual Technology
.
SOLUTION_2:
Open terminal or cmd (Run as admin) for Windows, and run SC START VBOXDRV. If it says the service is already running then try SC STOP VBOXDRV and then SC START VBOXDRV.
RealScar solution helped me in addition to other commands and it worked in Ubuntu 20.04, too.
I had the problem initially indicated (VirtualBox unable to find a registered machine). I was getting no results after typing sudo VBoxManage -nologo list vms, so I manually registered the existing machine typing sudo vboxmanage registervm /home/user/VirtualBox\ VMs/machinenamefolder/machinename.vbox. It worked great.
Note: I was creating a Cuckoo Malware Sandbox Analysis.
REASON: In the above case its uuid-mismatch (w.r.t. what is pointed to what is generated). Typically case of either improper edit of the vbox config files or accidentally deleted images/configs associated.
SOLUTION:
As FIRST Step: Correct the UUID-Mappings;
So, for example, the uuids can be regenerated to correct the mappings. (1)
Or otherwise, if attempted to edit .vbox/.vmdk/VirtualBox.xml files, the mappings should be corrected. (2)
As SECOND Step: Re-register the virtual-machine to the UI.
For example, if can't open the vm's from vbox gui or terminal. Remove the "inaccessible" entry from the GUI first". And then, open the folder of virtual machine and open file <machine name>.vbox with virtualbox and it will get registered. Provided, the uuid-mappings have been already resolved. Else follow the errors, while you attempt to register to make necessary changes in .vbox file.