I am presented with a program developed in c++ that implements a compute intensive algorithm that uses mpi to achieve better results. The executable has been tested on a single VM with 16 cores on Azure and the results have been satisfying. Now we need to test its performance and its scalability on 64 or 128 cores. As far as I know , a single VM can't employ more than 16 cores , so I think I need to implement a set of VMs that will execute the computation, but I don't know where to start with the deployment and the communication among the VMs. Any guidance or ideas would be appreciated.
Related
i have a newbie question in here, but i'm new to clouds and linux, i'm using google cloud now and wondering when choosing a machine config
what if my machine is too slow? will it make the app crash? or just slow it down
how fast should my vm be? in the image bellow
last 6 hours of a python scripts i'm running and it's cpu usage, it's obviously running for less than %2 of the cpu for most of it's time, but there's a small spike, should i care about the spike? and also, how much should my cpu usage be max before i upgrade? if a script i'm running is using 50-60% of the cpu most of the i assume i'm safe, or what's the max before you upgrade?
what if my machine is too slow? will it make the app crash? or just
slow it down
It depends.
Some applications will just respond slower. Some will fail if they have timeout restrictions. Some applications will begin to thrash which means that all of a sudden the app becomes very very slow.
A general rule, which varies among architects, is to never consume more than 80% of any resource. I use the rule 50% so that my service can handle burst traffic or denial of service attempts.
Based on your graph, your service is fine. The spike is probably normal system processing. If the spike went to 100%, I would be concerned.
Once your service consumes more than 50% of a resource (CPU, memory, disk I/O, etc) then it is time to upgrade that resource.
Also, consider that there are other services that you might want to add. Examples are load balancers, Cloud Storage, CDNs, firewalls such as Cloud Armor, etc. Those types of services tend to offload requirements from your service and make your service more resilient, available and performant. The biggest plus is your service is usually faster for the end user. Some of those services are so cheap, that I almost always deploy them.
You should choose machine family based on your needs. Check the link below for details and recommendations.
https://cloud.google.com/compute/docs/machine-types
If CPU is your concern you should create a managed instance group that automatically scales based on CPU usage. Usually 80-85% is a good value for a max CPU value. Check the link below for details.
https://cloud.google.com/compute/docs/autoscaler/scaling-cpu
You should also consider the availability needed for your workload to keep costs efficient. See below link for other useful info.
https://cloud.google.com/compute/docs/choose-compute-deployment-option
I've been tasking with creating a Minecraft server for about 500 players. I've never create a Minecraft server this large before and I think the best way to go about it is (how I've done all my other servers) with Google Cloud. Google Cloud has the following VM options:
I'm thinking about ~100MB per player to be on the safe side, so that's about 50 GB, so I'd say the n2-highmem-8 is probably a good VM to use. Is this overkill? Underkill?
Also, I know that a MC server can't use Multiple CPUs at the same time, so is it a waste paying for 8 Virtual CPUs?
Thanks!
For 500 players, I would recommend a minimum of 6 vCore and 32 GB or more.
You're right with Minecraft mainly using only one thread, so a high CPU Clock speed is most valuable. Also, I would recommend using PaperMC, it's a custom-made better performing Minecraft Server software. Here could you get different server versions if you do not want to use 1.15.2 Minecraft.
Maybe you should look around for different Servers, near your location. I bet you can find something less expensive and with higher Core clocks.
Minecraft has FREE servers in the launcher
Step1
Step2
A c5.2xlarge instance has 8 vCPU. If I run os.cpu_count() (Python) or std::thread::hardware_concurrency() (C++) they each report 8 on this instance. I assume the underlying hardware is probably a much bigger machine, but they are telling me what I have available to me, and that seems useful and correct.
However, if my ECS task requests only 2048 CPU (2 vCPU), then it will still get 8 from the above queries on a c5.2xlarge machine. My understanding is Docker is going to limit my task to only using "2 vCPU worth" of CPU, if other busy tasks are running. But it's letting me see the whole instance.
It seems like this would lead to tasks creating too many threads/processes.
For example, if I'm running 2048 CPU tasks on a c5.18xlarge instance, each task will think it has 72 cores available. They will all create way too many threads/processes overall; it will work but be inefficient.
What is the best practice here? Should programs somehow know their ECS task reservation? And create threads/processes according to that? That seems good except then you might be under-using an instance if it's not full of busy tasks. So I'm just not sure what's optimal there.
I guess the root issue is Docker is going to throttle the total amount of CPU used. But it cannot adjust the number of threads/processes you are using. And using too many or too few threads/processes is inefficient.
See discussion of cpu usage in ECS docs.
See also this long blog post: https://goldmann.pl/blog/2014/09/11/resource-management-in-docker/
There is a huge difference between virtualization technologies and containers. Having a clear understanding of these technologies will help. That being said an application should be configurable if you want to deploy it in different environments.
I would suggest creating an optional config which tells the application that it can only use certain number of cpu cores. If that value is not provided then it falls back to auto detect.
Once you have this option when defining ECS task you can provide this optional config, which will fix the problem you are facing.
I have been using amazon EC2 instances to run a C++ program. Its heaviest task is to launch a boost dijkstra algorithm. On my laptop (running on Ubuntu 15.04) and on the EC2 instances (Ubuntu 14.04) I had similar performances. A typical dijkstra would take 60ms. Interestingly enough, the type of EC2 instance would have low performance impact on a single Dijkstra run.
Now, I've just set up an OVH cloud server, runnning on Ubuntu 14.04. I have followed the same steps to install all the dependencies I need. The very same Dijkstra now takes 130ms to run.
Boost version is the same, as are my compiler options (I'm using -O3). I've tried different types of OVH instances (RAM oriented, CPU oriented). The timing remains unchanged.
I doubt that OVH cloud performance could be this bad. Any idea ideas about what I could have missed or tests that I could do to understand what is going on ?
Right now we are building a number of C++ apps for Win32 platform. We will be soon porting to Linux and then maybe more (32 and 64 bits for both).
What is the standard practice , do you use multiple hudson servers each on their own platform to do a build, or does the hudson service create VMs and do builds?
It is not clear to me the best practical way to do this.
Ideally I just want one box with a bunch of VMs running hudson, and then it kicks off builds as needed.
Is there a resource someone can point me to for this?
We use Hudson to manage C/C++ (GNU C, GNU C++, Watcom C) builds for multiple OSs. For us, software is built for Linux, Linux x64, QNX 4, and QNX6. The way we have it set up is:
1 x VM for the Hudson server, running Windows
4 x VMs, one for each slave type, so I have 4 Hudson slaves - 1 each for QNX4, QNX6 and Linux 32 and Linux 64. All of them are running on the same server, just as different VMs, and we have faced no problems. We build about a 100 projects, divided almost equally between the 4 system types.
You should not require any additional hardware. There is a Hudson plugin that works with VMWare VMs, to start them up and shut them down as required.
I hope that helps.
I've never used hudson for C++ but for what you are planning to do, it might make sense to look at the VMWare plugin and see if it will do what you want. I would recommend having only a single Hudson master if possible. What you most likely want to do is set up a VMWare machine image with a Hudson Slave process for each target environment then spawn a build in that Slave.
I have played with hudson in a multiple platform scenario a bit more than a year ago. I had one hudson server (which was ridiculously easy to setup) on some machine and separate build slaves for each of the platforms. I remember that for a while one of the build clients was in a VirtualBox on the machine that hosted the hudson server. (I think I had the server on a VM for a while, too.) I cannot remember there being any principle problem with this setup.
However, if you want to have several virtual build machines building on the same physical machine I think you'd need a very powerful machine for that. C++ compilation takes quite an amount of resources and, IIRC, when hudson starts a build, it starts it on all platforms at the same time.
Note that there need not be any relation between the server that's running Hudson and the slave machines that are building your software apps. Due to the magic of Java, you can connect the disparate slave machines to the master using JNLP. (one example) So, whether they are physical or virtual machines, you can have one running Windows, another Linux; one 32-bit, another 64-bit; etc -- whatever your apps require. As long as they all have the JRE installed they can connect to the Hudson master and report the status of the builds.