NOTE: This is on Windows.
I have an application that is started as pm2 start index.js --name dvc -- config.json. Then, I started a new command window to monitor the application pm2 monit. To test the application, I am using the Runner option in Postman where, the number of iterations is set to 1000 with a delay of 0 ms.
In pm2 monit window, the CPU % remains between 0 and 11%. In Task Manager, the node.exe process shows CPU % in 20s. The Process Explorer shows the CPU utilisation close to values as reported by pm2 monit. So, I am not able to conclude exactly what is the CPU utilisation.
Can you please advise?
I would recommend looking into Windows Performance Monitor instead, it exposes more precise counters
Start Performance Monitor (i.e. type perfmon in "Search" or "Run" boxes and click "Enter")
Add a new Counter (click green plus sign)
Choose Process from the "Available counters" and search for node
You should see charts for different counters (including, but not limited to CPU usage)
Be aware of following:
On multi-core processor systems you might need to monitor CPU usage for all cores in order to ensure that your application can be paralellised
Your 1000 iterations don't really create any load as Postman waits for previous response prior to sending a new request therefore you always have only 1 request processed by your system which might be even cached. If you would like to load test your application I would recommend considering another tool capable of sending requests in multithreaded fashion, for example Apache JMeter would be a reasonable choice. Check out REST API Testing - How to Do it Right article for instructions on seting up JMeter for an API load testing.
I am familiar with Docker, Rkt and LXD, but if I did not have the ability to install all these tools, what would be the basic mechanisms to provide isolation of CPU, memory and Disk for a particular process?
CPU - I want to say that only 1 socket of the two is usable by this process
Memory - I don't want this process to use more than 10GB memory
Disk - I don't want the process to use more than 100GB of disk and have visibility (ls should not list it) of files that are not created by this process
I think installing Docker, Rkt and what-not is very heavy weight solution for something basic that I am trying to accomplish
Is cgroups the underlying API I should tap into to get what I need? If so, is there a good book to learn about CGroups
I am running on EC2 - RHEL and Ubuntu both.
See man page for cgroups(7) for introduction, the full documentation of cgroup interface is maintained in linux kernel:
cgroup v1
cgroup v2
On top of that, on a distribution with systemd and cgroup v2 interface, cgroup features should be used via systemd and not directly. See also man page for systemd.resource-control.
For distribution specific information, see:
RHEL 6 Resource Management Guide
RHEL 7 Resource Management Guide
Quick answers to your questions
I want to say that only 1 socket of the two is usable by this process
This could be done via cpuset controller from cgroup v1 (both on RHEL 6 and RHEL 7).
I don't want this process to use more than 10GB memory
See memory controller of cgroup v1 interface or MemoryLimit of systemd resource control interface.
I don't want the process to use more than 100GB of disk
This is out of cgroups area of control, use disk quotas instead.
have visibility (ls should not list it) of files that are not created by this process
This is out of cgroups functionality, use either filesystem access right, filesystem namespaces or PrivateTmp systemd service option, depending on your use case.
How do I enable CPU hot add and Remove for this virtual machine option?Are they any OS specific configuration required or NUMA architecture is required?
Yes, It is specific.
Linux: Newer Linux kernels have support for CPU hot add and CPU hot remove. To verify this, check the documentation for your Linux distribution in the documentation directory of the distribution source code. The documentation contains directions for special boot time switches related to CPU Hot Plug, as well as how to dynamically bring CPUs on and offline.
Windows: Windows Server 2012 (Standard and Datacenter Edition) and Windows Server 2008 Datacenter Edition support CPU hot add, but not CPU hot remove.
Note: Windows Server 2008 Standard and Enterprise Editions do not support CPU hot add.
Check and conform the virtual machine's are using hardware version 7 or later.
For information regarding hot add Refer this links.
VMware KB
Hot Plug Settings
Window System 7x64 SP1
VM Workstation 11
When virtualization engine preferred mode is explicitly enabled with Intel VT-x or AMD-V, the following message is displayed during boot of the VM:
"vmware virtualized intel vt-x/ept requires a preferred mode of Intel
VT-x/ept reverting to automatic?"
The Intel Processor Identification Utility (PIU) reports Virtualization technologies capabilities. In the BIOS, it does not have an option to explicitly enable VT-x.
Q: Does the PIU test indicate the VT-x is enabled implicitly in the BIOS
When booting with virtualization engine preferred mode is explicitly enabled with Automatic. The settings appear with both virtualize VT-x/EPT and CPU performance grayed out:
When booting with virtualization engine preferred mode is explicitly enabled with Automatic. The settings appear with both virtualize VT-x/EPT and CPU performance grayed out:
Q: I am uncertain what this means?
Q: Is there a way (a service, process, etc) to determine if Virtualization technologies are being enabled for the VM?
I have noticed that our VMWare VMs often have the incorrect time on them. No matter how many times I reset the time they keep on desyncing.
Has anyone else noticed this? What do other people do to keep their VM time in sync?
Edit: These are CLI linux VMs btw..
If your host time is correct, you can set the following .vmx configuration file option to enable periodic synchronization:
tools.syncTime = true
By default, this synchronizes the time every minute. To change the periodic rate, set the following option to the desired synch time in seconds:
tools.syncTime.period = 60
For this to work you need to have VMWare tools installed in your guest OS.
See http://www.vmware.com/pdf/vmware_timekeeping.pdf for more information
according to VMware's knowledge base, the actual solution depends on the Linux distro and release, in RHEL 5.3 I usually edit /etc/grub.conf and append this parameters to the kernel entry: divider=10 clocksource=acpi_pm
Then enable NTP, disable VMware time synchronization from vmware-toolbox and finally reboot the VM
A complete table with guidelines for each Linux distro can be found here:
TIMEKEEPING BEST PRACTICES FOR LINUX GUESTS
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427
I'll answer for Windows guests. If you have VMware Tools installed, then the taskbar's notification area (near the clock) has an icon for VMware Tools. Double-click that and set your options.
If you don't have VMware Tools installed, you can still set the clock's option for internet time to sync with some NTP server. If your physical machine serves the NTP protocol to your guest machines then you can get that done with host-only networking. Otherwise you'll have to let your guests sync with a genuine NTP server out on the internet, for example time.windows.com.
Something to note here. We had the same issue with Windows VM's running on an ESXi host. The time sync was turned on in VMWare Tools on the guest, but the guest clocks were consistently off (by about 30 seconds) from the host clock. The ESXi host was configured to get time updates from an internal time server.
It turns out we had the Internet Time setting turned on in the Windows VM's (Control Panel > Date and Time > Internet Time tab) so the guest was getting time updates from two places and the internet time was winning. We turned that off and now the guest clocks are good, getting their time exclusively from the ESXi host.
In my case we are running VMWare Server 2.02 on Windows Server 2003 R2 Standard. The Host is also Windows Server 2003 R2 Standard. I had the VMware Tools installed and set to sync the time. I did everything imaginable that I found on various internet sites. We still had horrendous drift, although it had shrunk from 15 minutes or more down to the 3 or 4 minute range.
Finally in the vmware.log I found this entry (resides in the folder as the .vmx file):
"Your host system does not guarantee synchronized TSCs across different CPUs, so please set the /usepmtimer option in your Windows Boot.ini file to ensure that timekeeping is reliable. See Microsoft KB http://support.microsoft.com/kb... for details and Microsoft KB http://support.microsoft.com/kb... for additional information."
Cause: This problem occurs when the computer has the AMD Cool'n'Quiet technology (AMD dual cores) enabled in the BIOS or some Intel multi core processors. Multi core or multiprocessor systems may encounter Time Stamp Counter (TSC) drift when the time between different cores is not synchronized. The operating systems which use TSC as a timekeeping resource may experience the issue. Newer operating systems typically do not use the TSC by default if other timers are available in the system which can be used as a timekeeping source. Other available timers include the PM_Timer and the High Precision Event Timer (HPET).
Resolution: To resolve this problem check with the hardware vendor to see if a new driver/firmware update is available to fix the issue.
Note The driver installation may add the /usepmtimer switch in the Boot.ini file.
Once this (/usepmtimer switch) was done the clock was dead on time.
This documentation solved this problem for me.
The CPU speed varies due to power saving. I originally noticed this because VMware gave me a helpful tip on my laptop, but this page mentions the same thing:
Quote from : VMWare tips and tricks
Power saving (SpeedStep, C-states, P-States,...)
Your power saving settings may interfere significantly with vmware's performance. There are several levels of power saving.
CPU frequency
This should not lead to performance degradation, outside of having the obvious lower performance when running the CPU at a lower frequency (either manually of via governors like "ondemand" or "conservative"). The only problem with varying the CPU speed while vmware is running is that the Windows clock will gain of lose time. To prevent this, specify your full CPU speed in kHz in /etc/vmware/config
host.cpukHz = 2167000
VMware experiences a lot of clock drift. This Google search for 'vmware clock drift' links to several articles.
The first hit may be the most useful for you: http://www.fjc.net/linux/linux-and-vmware-related-issues/linux-2-6-kernels-and-vmware-clock-drift-issues
When installing VMware Tools on a Windows Guest, “Time Synchronisation” is not enabled by default.
However – “best practise” is to enable time synch on Windows Guests.
There a several ways to do this from outside the VM, but I wanted to find a way to enable time sync from within the guest itself either on or after tools install.
Surprisingly, this wasn’t quite as straightforward as I expected.
(I assumed it would be posible to set this as a parameter / config option during tools install)
After a bit of searching I found a way to do this in a VMware article called “Using the VMware Tools Command-Line Interface“.
So, if time sync is disabled, you can enable it by running the following command line in the guest:
VMwareService.exe –cmd “vmx.set_option synctime 0 1″
Additional Notes
For some (IMHO stupid) reason, this utility requires you to specify the current as well as the new value
0 = disabled
1 = enabled
So – if you run this command on a machine which has this already set, you will get an error saying – “Invalid old value“.
Obviously you can “ignore” this error when run (so not a huge deal) but the current design seems a bit dumb.
IMHO it would be much more sensible if you could simply specify the value you want to set and not require the current value to be specified.
i.e.
VMwareService.exe –cmd “vmx.set_option synctime <0|1>”
In Active Directory environment, it's important to know:
All member machines synchronizes with any domain controller.
In a domain, all domain controllers synchronize from the PDC Emulator (PDCe) of that domain.
The PDC Emulator of a domain should synchronize with local or NTP.
It's important to consider this when setting the time in vmware or configuring the time sync.
Extracted from: http://www.sysadmit.com/2016/12/vmware-esxi-configurar-hora.html
I added the following job to crontab. It is hacky but i think should work.
*/5 * * * * service ntpd stop && ntpdate pool.ntp.org && service ntpd start
It stops ntpd service updates from service and starts ntpd again