We are a startup company and doesnt have invested yet in HW resources in order to prepre our dev and testing environment. The suggestion is to buy a high end server, install vmware ESX and deploy mutiple VMs for build, TFS, database, ... for testing,stging and dev enviornment.
We are still not sure what specs to go with e.g. RAM, whether SAN is needed?, HD, Processor, etc..?
Please advice.
You haven't really given much information to go on. It all depends on what type of applications you're developing, resource usage, need to configure different environments, etc.
Virtualization provides cost savings when you're looking to consolidate underutilized hardware. If each environment is sitting idle most of the time, then it makes sense to virtualize them.
However if each of your build/tfs/testing/staging/dev environments will be heavily used by all developers during the working day simultaniously then there might not be as many cost savings by virtualizing everything.
My advice would be if you're not sure, then don't do it. You can always virtualize later and reuse the hardware.
Your hardware requirements will somewhat depend on what kind of reliability you want for this stuff. If you're using this to run everything, I'd recommend having at least two machines you split the VMs over, and if you're using N servers normally, you should be able to get by on N-1 of them for the time it takes your vendor to replace the bad parts.
At the low-end, that's 2 servers. If you want higher reliability (ie. less downtime), then a SAN of some kind to store the data on is going to be required (all the live migration stuff I've seen is SAN-based). If you can live with the 'manual' method (power down both servers, move drives from server1 to server2, power up server2, reconfigure VMs to use less memory and start up), then you don't really need the SAN route.
At the end of the day, your biggest sizing requirement will be HD and RAM. Your HD footprint will be relatively fixed (at least in most kinds of a dev/test environment), and your RAM footprint should be relatively fixed as well (though extra here is always nice). CPU is usually one thing you can skimp on a little bit if you have to, so long as you're willing to wait for builds and the like.
The other nice thing about going all virtualized is that you can start with a pair of big servers and grow out as your needs change. Need to give your dev environment more power? Get another server and split the VMs up. Need to simulate a 4-node cluster? Lower the memory usage of the existing node and spin up 3 copies.
At this point, unless I needed very high-end performance (ie. I need to consider clustering high-end physical servers for performance needs), I'd go with a virtualized environment. With the extensions on modern CPUs and OS/hypervisor support for them, the hit is not that big if done correct.
This is a very open ended question that really has a best answer of ... "It depends".
If you have the money to get individual machines for everything you need then go that route. You can scale back a little on the hardware with this option.
If you don't have the money to get individual machines, then you may want to look at a top end server for this. If this is your route, I would look at a quad machine with at least 8GB RAM and multiple NICs. You can go with a server box that has multiple hard drive bays that you can setup multiple RAIDS on. I recommend that you use a RAID 5 so that you have redundancy.
With something like this you can run multiple VMWare sessions without much of a problem.
I setup a 10TB box at my last job. It had 2 NICs, 8GB, and was a quad machine. Everything included cost about 9.5K
If you can't afford to buy the single machines then you probably are not in a good position to start re-usably with virtualisation.
One way you can do it is take the minimum requirements for all your systems, i.e. TFS, mail, web etc., add them all together and that will give you an idea of half the minimum server you need to host all those systems. Double it and you be near what will get you buy, if you have spare cash double/triple the RAM. Most OSes run better with more RAM to particular ceiling. Think about buying expandable storage of some kind and aim for half populated to start with which will keep the initial cost/GB and make for some expansion at lower cost in the future.
You can also buy servers which take multiple CPUs but only put in the minimum amount of CPUs. Also go for as many cores on a CPU as you can get for thermal, physical and licensing efficiency.
I appreciate this is a very late reply but as I didn't see many ESX answers here I wanted to post a reply though my post equally relates to Hyper-V etc.
Related
Not sure where to post this question, but I was looking at DO and it seemed pretty compelling.
My coworker and I need new computers pretty soon at the office just to improve quality of life for when we do analysis or wrangling data on large files/queries.
I could purchase two physical computers which will become outdated (the options we have already look fairly old actually, we are stuck with Dell), or perhaps we can 'move to the cloud'. The upfront cost would be several thousand dollars and we would wait for a few months before receiving them if we chose the physical machines.
A challenge we have had is sharing tasks (like a script that cleans data and loads it into the database) and sharing the raw data files (prior to adding to the database). It would be nice to centralize all of this stuff because there has been a growing fragmentation and it makes it challenging to travel for long periods of time in case something breaks.
Is it possible to centralize some files and scripts on a DO instance and then both of us can access that machine and add new scripts, pull data, or make alterations as needed? Our computers at the office would just be for email and stuff, all of the heavy lifting could be done with DO.
I have about 4 days to load 4 million rows of data, where the stored procedures appear to take about 500ms/row, so I am suggesting we temporarily repurpose two computers, one as the SQL Server and one to feed it. Each machine is a 4 cpu, 2 core/cpu computer with tons of memory.
There are two competing ideas on how to load the data.
on the import computer maintain using a VM, which limits me to 4 cores of use.
ditch the VM for the load and use all 8 cores for loading.
The database will be on the actual computer, no VM.
I will get one shot at showing that using the VM will considerably impact performance, but that is just a gut feeling. I need to prove it, so I am going to suggest we do a test load with a smaller dataset, so rather than loading about 12G data we will do perhaps only 6G, and get numbers.
But, since I have one shot at this, I want to get a reasonable guess at how many threads can be used in both environments.
My guess is 4 threads/core, since database transactions involve a great deal of waiting time for threads, but I don't know if that would be too many threads on the VM. I am not certain how to determine what is a reasonable number of threads to suggest.
The limiting factor may not be the computer that has the program that loads the data, it may be the database computer will actually be the limiting factor, and 32 concurrent database connections may be too much for an 8 core machine.
So, is my guess of 4 threads per core reasonable in both environments?
The OS on the computers is Windows Server 2008 R2 and the VM would be using Hyper-V.
UPDATE:
In case it matters, I am using .NET 4, using the parallel framework, so I adjust the number of concurrent threads by changing it in my configuration file, so I use PLINQ heavily.
UPDATE 2:
I had an idea today to speed things up, where we will use one 4 core VM to benchmark but the SQL Server will use a 24G virtual drive (ram drive), since the computer has 48G RAM, and the database will be completely in memory. This way I should be able to feed it with several threads as fast as the feeder program can. If that isn't fast enough then we will use all 12 cores to feed it, but they expect it won't be needed. So, I should be able to hopefully load all the data in 1 day instead of a bit over a month, but, once I get some benchmarks I will let everyone know. I may see about using a smaller desktop to feed it to see if there is a penalty by using a VM, but there will only be one VM on this box, so 8 cores will be unused. The OS on the Hyper-V box is Windows Core, btw.
There are so many variables involved that I would say your guess is as good as any. You have thought it out reasonably and testing will be your best bet to see if you need to adjust for the big run.
Make a small test, and test with 1, 2, 3, 4, 5, 6 threads per core. Graph it out. Should show pretty clearly...
My company is at the very end of development process of some (awesome:)) web application. This app will be available as a online service for (hopefully) some significant number of users. This is our biggest django release so far and as we are preparing to release some question about deployment have to be answered.
Q1: how to determine required server parameters for predicted X number of users/Y hits per minute or other factor?
Q2: what hosting solution (shared/vps/dedicated) is worth considering?
Q3: what optimizations can be done at a first place?
I know that this is very subjective and dependent of size of a site, code quality and other factors but I'm very interested in your experiences with django (and not only django) deployment. Any hints, links, advices are kindly welcome. Thanks in advance.
What hosting solution you want to have depends also on how much you want to take of your server yourself (from upgrades etc to backup...), and you should decide if you want to have the responsibility or leave it to someone else.
I think you can only really determine the necessary requirements and bottlenecks in your applications through testing with the estimated load! Try to simulate as many requests.... as you expect - think about caching (where memcached is the best option you have)! If you try to cache things one great tool is the django debug toolbar (http://github.com/robhudson/django-debug-toolbar) which can show you also much about how many database hits you have (dont take the times it shows for that for granted, but analyse them and keep an eye on the number of hits) and eg. how many templates are rendered....
If your system grows, you can first of all think about serving your static media files from another location! Coming to the web server I have made great experiences using lighttpd instead of the fat apache, but I guess you have to evaluate that for yourself!
Also take in consideration what database backend to use, in shared envionments there's in most cases a bigger load on the mysql than on the postgres servers, but also evaluate what works best for you!
You can get some guesses here, but to get a halfway reasonable performance estimate you have to measure the performance of your application yourself. You should then be able to roughly extrapolate the performance on different hardware.
Most of the time the bottleneck is the database, you should get enough RAM to keep it in memory if possible.
"Web application" can encompass so many different things, we can really do no more than guess here.
As for optimization, if it fits to your needs implement some caching (e.g. with memchached), that can give you huge speed improvements.
There are different "Application memory" options (like 80MB...200MB) in django-friendly hosting called webfaction and I'm confused deciding which one I should buy.
Could someone please walk me through the ideas on how to figure out how much memory my project might require (excluding operating system, the main apache server and the database servers memory requirements)? I understand in theory I'll need to perform some kind of load testing, but thought there might be ways to calculate that in advance with some simple/relatively easy understandable approach.
I don't know how hard they enforce application memory usage limit, and another question is: what will happen if more users came to the site and more threads started than what I expected? Will the application crash? Or will delays just become uncomfortable?
And - no, application is not ready yet (I can't measure anything right now). Development environment if it matters is Winodows 7, 64-bit. Hosting itself is some kind of Linux I think.
(Sorry if it's not a stackoverflow question.)
Sorry, but until you have the application completely developed, you can't say anything about the kind of memory it'll use. I recommend that you take their "lowest" plan, and renew to it to fit your needs, or still better: get hosting after you finish developing the application.
On the other hand, if you had the application ready, you could just run it in Apache with your host's config and some sample data to get a rough estimate...
I agree that you can' tell much before your app is ready.
As a vague estimation consider that your host is supposed to be "django-friendly" so some "basic" application should run without problems. Try and upgrade later if that's possible easily.
Also consider the type of data that is processed with your app, I eg. ran into troubles once when I had to process really large image uploads that made the whole site crash.
Also keep in mind if you need some ram for additional processes eg. memcache!
Webfaction are indeed a Django-friendly host, and your application will certainly not crash if it starts needing more memory than you have paid for. What will happen is that you will be allowed to use small amounts of extra memory, but if you consistently go over the limit they will send you a polite email requesting that you either reduce the load or pay for more.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 years ago.
Improve this question
Greetings!
I'd like to build an apache web server, running on debian lenny.
It will primarily be used for hosting a web-shop, so it should have some light db i/o and lots of image serving (item previews/thumbs/etc...).
It's tough to put a finger on the exact number of concurrent requests that I'll get hit with, but I'd say that a non-professional setup should be enough to handle them.
By non-professional I mean that I don't need to invest into purchasing blades, a rack or something of the like. Just regular desktop PC tweaked for web server performance.
Which exposes my current problem: I have no idea whatsoever what kind of a machine I should be looking for.
If I'd like to build a gaming rig, no problem - there are at least a million sites out there with performance benches, from bleeding edge graphic card reviews to flat panel LCD contrast/response time charts. But when it comes to trying to find reccomendations for a web server based build, I'm having a hard time finding a good RECENT review.
So, at least I've managed to gather this so far - these are the priorities I should be attending to:
1) Lots of memory (preferably fast)
2) A pair of fast HDDs
3) As many cores as I can get
4) As fast processor as I can get
5) A MB with good I/O
So, memory and HDDs aren't that big of a deal, you can't go wrong here (I guess).
With the RAM prices these days, it's pretty affordable to pump 8+ Gb into a machine.
The only question here is, would it be worth it to buy a tiny (<=32 Gb) SSD and place all my web stuff and OS onto it. My entire web server is just a couple of megs in size + the database will fit really neatly onto it with space to boot.
As for the graphics card, I'll just plug in any old PCI Ex card I can whip up, and the same goes for any peripherals. I don't need a display of any kind - I'll be logging in remotely for most of the time.
OK - and now for the most important question: Which Proc and MB to buy.
As far as I've gathered - it would be better to have 10 cores running at 100 Mhz each than only one running at 2 Ghz, taking the nature of the machine into consideration.
So I'll most likely have to get a quad core, right? The question is which... :/
For there are several affordable... My budget is around US $800. This is, again, for just the proc, the MB, and the memory. I have the HDDs. If I take a small SSD, add $100 to that budget.
AMD Phenom or Intel Core 2? Which MB to go with it? I'm totally lost here.
If this will start an AMD vs. Intel flame war, I'm truly sorry, for this is not my intention - but if you could at least point me to a good recent review for a web server build I would ge grateful.
On the one hand you say you don't need that much performance but on the another you are talking about adding as many cores as you can. A quad core CPU of either AMD or Intel will be more than sufficient. It gets into the category of "religious war" but i prefer the Intel chips; I usually buy Xeon processors. As far as SSD, I wouldn't bother. Look into a good RAID setup with a 3Ware controller; either RAID 1+0 or RAID 5 (obviously, there will be a religious anti-RAID5 crowd, though I prefer it .. at least until RAID 6 is more widespread). As much memory as you can afford is ideal, although anything more than 8 is probably overkill from what you have said. Probably the main departure from what you have already listed is that I wouldn't even bother with the SSD. Depending on your usage patterns, you may actually hurt performance with it and any benefits for your use cases would not be worth the costs. Wait for the research to catch up for SSD to really be beneficial in terms of performance. :)
If this is a business server, I recommend buying one pre-configured from IBM, Dell, or whatever major manufactuer is your preference (I prefer IBM).
This is really a stretch for the "right" kind of question for SO. Only in degrees though "implementation."
Pre-configured "Server" machines can often be more cost-beneficial. But, if you'd still prefer to build your own...
Considering just your budget ($800) for MB, Proc, and Memory...
RAM - DDR2 800 ($200/4GB, and cheaper)
MB - 1333/1066MHz FSB ($250)
CPU - Dual Core ($150)
Quad Core can still be too expensive for the benefit -- but, that's up to you to judge.
But, follow the links, and use the Advanced Search to cross out unnecessary features, and you should be able to reduce the list of items fairly easily.
Have you considered shared, dedicated, or virtual hosting? If I were you, I'd go with SliceHost for the virtual server, then use Amazon S3 for serving up images and other large static files. The combination has worked well for me in the past. I've found that, especially when it comes to hosting, don't take on more responsibility than you absolutely have to.
I use MediaTemple for my websites. They have a lot of professional organizations hosted on there servers. I'd probably go with them if I were you.
My dad thought the server route would be easy and we found out differently the hard way. If you don't have a friend or an employee that really knows what he's doing, I'd be careful. Anyways, good luck.
If you're not planning on running the next Amazon, I'd say that your choice of CPU/chipset is irrelevant. Find a motherboard with the features you need (4+ RAM slots, plenty of SATA headers, etc) that suits your budget and then buy a upper midrange multicore CPU to suit. Get a PCI express RAID card and a meaty UPS too.
Get a vanilla hard drive for the OS, and a pair of fast drives (WD Velociraptors, etc) and put them in RAID 1 for the webserver for redundancy.
Then, after a year or so or restarting the server every other day, migrate everything to a hosting company.