I am familiar with Docker, Rkt and LXD, but if I did not have the ability to install all these tools, what would be the basic mechanisms to provide isolation of CPU, memory and Disk for a particular process?
CPU - I want to say that only 1 socket of the two is usable by this process
Memory - I don't want this process to use more than 10GB memory
Disk - I don't want the process to use more than 100GB of disk and have visibility (ls should not list it) of files that are not created by this process
I think installing Docker, Rkt and what-not is very heavy weight solution for something basic that I am trying to accomplish
Is cgroups the underlying API I should tap into to get what I need? If so, is there a good book to learn about CGroups
I am running on EC2 - RHEL and Ubuntu both.
See man page for cgroups(7) for introduction, the full documentation of cgroup interface is maintained in linux kernel:
cgroup v1
cgroup v2
On top of that, on a distribution with systemd and cgroup v2 interface, cgroup features should be used via systemd and not directly. See also man page for systemd.resource-control.
For distribution specific information, see:
RHEL 6 Resource Management Guide
RHEL 7 Resource Management Guide
Quick answers to your questions
I want to say that only 1 socket of the two is usable by this process
This could be done via cpuset controller from cgroup v1 (both on RHEL 6 and RHEL 7).
I don't want this process to use more than 10GB memory
See memory controller of cgroup v1 interface or MemoryLimit of systemd resource control interface.
I don't want the process to use more than 100GB of disk
This is out of cgroups area of control, use disk quotas instead.
have visibility (ls should not list it) of files that are not created by this process
This is out of cgroups functionality, use either filesystem access right, filesystem namespaces or PrivateTmp systemd service option, depending on your use case.
Related
I recently tried to bring up a Kubernetes cluster in AWS using kops. But when the worker node (Ubuntu 20.04) started, a docker load process on it kept getting OOMkilled even when it has enough memory (~14GiB). I tracked down the issue being I set kubelet's memory reservation too small (--kube-reserved=memory=100Mi...).
So now I have two questions related to the following paragraph in the documentation:
kube-reserved is meant to capture resource reservation for kubernetes system daemons like the kubelet, container runtime, node problem detector, etc.
https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved
First, I interpreted the "reservation" as "the amount of memory guaranteed", similar to the concept of a pod's .spec.resource.requests.memory. However, it seems like the flag acts like a limit as well? Does this mean Kubernetes intend to manage Kubernetes system daemons with "guaranteed" QoS class concept?
Also, my container runtime, docker, does not seem to be in /kube-reserved cgroup, instead, it is in /system.slice:
$ systemctl status $(pgrep dockerd) | grep CGroup
CGroup: /system.slice/docker.service
So why is it getting limited by /kube-reserved? It is not even kubelet talking to docker through CRI, but just my manual docker load command.
kube-reserved is a way to protect Kubernetes system daemons (which includes the Kubelet) from running out of memory should the pods consume too much. How is this achieved? The pods are limited by default to an "allocatable" value, equal to the memory capacity of the node minus several flag values defined in the URL you posted, one of which is kube-reserved. Here's what this looks like for a 7-GiB DS2_v2 node in AKS:
But it's not always the Kubernetes system daemons that have to be protected from either pods or even OS components consuming too much memory. It can very well be the Kubernetes system daemons that could consume too much memory and start affecting the pods or other OS components. To protect against this scenario, there's an additional flag defined:
To optionally enforce kube-reserved on kubernetes system daemons,
specify the parent control group for kube daemons as the value for
--kube-reserved-cgroup kubelet flag.
With this new flag in place, should the aggregated memory use of the Kubernetes system daemons exceed the cgroup limit, then the OOM killer will step in and terminate one of their processes. To apply this to the picture above, with the --kube-reserved-cgroup flag specified, the Kubernetes system daemons are prevented from going over 1,638 MiB.
I'm still cheap.
I have a software development environment which is a bog-standard Ubuntu 11.04 plus a pile of updates from Canonical. I would like to set it up such that I can use an Amazon EC2 instance for the 2 hours per week when I need to do full system testing on a server "in the wild".
Is there a way to set up an Amazon EC2 server image (Ubuntu 11.04) so that whenever I fire it up, it starts, automatically downloads code updates (or conversely accepts git push updates), and then has me ready to fire up an instance of the application server. Is it also possible to tie that server to a URL (e.g ec2.1.mydomain.com) so that I can hit my web app with a browser?
Furthermore, is there a way that I can run a command line utility to fire up my instance when I'm ready to test, and then to shut it down when I'm done? Using this model, I would be able to allocate one or more development servers to each developer and only pay for them when they are being used.
Yes, yes and more yes. Here are some good things to google/hunt down on SO and SF
--ec2 command line tools,
--making your own AMI's from running instances (to save tedious and time consuming startup gumf),
--route53 APIs for doing DNS magic,
--ubunutu cloud-init for startup scripts,
--32bit micro instances are your friend for dev work as they fall in the free usage bracket
All of what James said is good. If you're looking for something requiring less technical know-how and research, I'd also consider:
juju (sudo apt-get install -y juju). This lets you start up a series of instances. Basic tutorial is here: https://juju.ubuntu.com/docs/user-tutorial.html
I need to find out the CPU utilization of a service DLL. I have looked in existing samples and we can find CPU utilization for processes.
I think DLL will be loaded by services.exe. So is it possible to find out CPU utilization by DLL.
I am working in C++ on the Windows platform.
Make a copy of svchost.exe and call it dbgsrvc.exe; then, go into the service entry in the registry (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog for example), and change the ImagePath to use dbgsrvc instead of services. That way, you've isolated your service into its own process so you can get perf counters on it.
The only performance counters available in Windows are at the process level. What do you mean by service DLL? The only service.DLL I know of is a trojan virus.
But are you talking about generic service DLLs or a specific file?
Try this:
Use Perfmon to log all counters for the process object.
Before or after your test, run tasklist /svc from a command console. Look through the output for the svchost.exe that's hosting your service. Note the PID of this process.
After the test, use Perfmon to load your log file, the add the process measurements for the process in step 2.
You should monitor the process hosting the dll.
We have 3 identical HP DL380 G5 server here, one of them is running vmware-server with one VM running on it.
I've begun the process to migrate these systems to be running ESXi (the $0, "embedded" system); two of the physical machines will have %99.99 of the time exactly 1 VM, the other will have 2.
For this, the major advantage I get Disaster Recovery ability. Our tape backup system doesn't have a "bare metal" ability. I can manually copy VM images to a different server, however. Even if they are months old, they provide pretty-close-to-instant up, further recovery they would be from tape.
Being the free version, I don't get the VMWare "consolidated backup" or VMotion. And I need to do per-physical machine management. But the ESXi takes 32MB of disk, and it specifically supports the server.
With that in mind, is there any reason not to always use ESXi, if the hardware supports it? Even if you only are planning on running 1 VM on that hardware?
Well, in your case ESXi is the better choice. There are cases where you want to use VMware Server but not really for this case. This is what ESXi is for. For instance, I use VMware Server on top of my development OS so I could do testing and use different distro's etc. I wouldn't do VMware Server for a production server like you are describing, but ESXi would be the best choice.
Is it an excellent idea to virtualize the whole OS to get the ability to make backups? NO! its not... Damn hype to virtualize without the real need for it.
There are free alternatives to make backups of pretty much any OS, image or archive of your choice.
To be more precise, XSIBackup will allow you to hot backup any ESXi edition from version 5.1 and up, it backs up the guest OS while it is running, and can even transfer it to a secondary ESXi box via IP and leave it ready to be switched on:
https://33hops.com/xsibackup-vmware-esxi-backup.html
Two RAID volumes, VMware kernel/console running on a RAID1, vmdks live on a RAID5. Entering a login at the console just results in SCSI errors, no password prompt. Praise be, the VMs are actually still running. We're thinking, though, that upon reboot the kernel may not start again and the VMs will be down.
We have database and disk backups of the VMs, but not backups of the vmdks themselves.
What are my options?
Our current best idea is
Use VMware Converter to create live vmdks from the running VMs, as if it was a P2V migration.
Reboot host server and run RAID diagnostics, figure out what in the "h" happened
Attempt to start ESX again, possibly after rebuilding its RAID volume
Possibly have to re-install ESX on its volume and re-attach VMs
If that doesn't work, attach the "live" vmdks created in step 1 to a different VM host.
It was the backplane. Both drives of the RAID1 and one drive of the RAID5 were inaccessible. Incredibly, the VMware hypervisor continued to run for three days from memory with no access to its host disk, keeping the VMs it managed alive.
At step 3 above we diagnosed the hardware problem and replaced the RAID controller, cables, and backplane. After restart, we re-initialized the RAID by instructing the controller to query the drives for their configurations. Both were degraded and both were repaired successfully.
At step 4, it was not necessary to reinstall ESX; although, at bootup, it did not want to register the VMs. We had to dig up some buried management stuff to instruct the kernel to resignature the VMs. (Search VM docs for "resignature.")
I believe that our fallback plan would have worked, the VMware Converter images of the VMs that were running "orphaned" were tested and ran fine with no data loss. I highly recommend performing a VMware Converter imaging of any VM that gets into this state, after shutting down as many services as possible and getting the VM into as read-only a state as possible. Loading a vmdk either elsewhere or on the original host as a repair is usually going to be WAY faster than rebuilding a server from the ground up with backups.