I need to host a service with rest-api on a server which does below listed tasks:
Download and upload files in s3 bucket
Run some cpu intensive computations
Return json response
I know an ec2 instance will be a better approach to host my service but given price differences between workspace and ec2 instance, I am exploring this route. Are there any limitations on amazon workspace that might prevent me from using them for my use case?
I came across ngrok which I believe can help me direct requests over the internet to my workspace local server.
Has anyone played around with it and could add some suggestion?
AWS terms of service do not allow you to do that I’m afraid. See section 36 on workspaces.
http://aws.amazon.com/service-terms/
36.3. You and End Users may only use the WorkSpaces Services for an End User’s personal or office productivity. WorkSpaces are not meant to accept inbound network connections, be used as server instances, or serve web traffic or your network traffic. You may not reconfigure the inbound network connections of your WorkSpaces. We may shut down WorkSpaces that are used in violation of this Section or other provisions of the Agreement.
I suggest you use an r5a.xlarge for the lowest cost 32GB RAM instance type (it’s AMD processor is cheaper than r5 on intel). Investigate whether spot instances would work if your state persists on S3 and not in the local instance, otherwise if you need it for at least a year reserved instances are discounted over on demand pricing.
Related
I recently start using GCP but i have one thing i can't solve.
I have: 1 VM + 1 DB Instance + 1 LB. DB instance allow only conections from the VM IP. bUT THE VM IP allow traffic from all ip (if i configure the firewall to only allow CloudFlare and LB IP's the website crash and refuse conections).
Recently i was under attack, i activate the Cloudflare ddos mode, restart all and in like 6 h the attack come back with the Cloudflare activate. Wen i see mysql conections bump from 20-30 to 254 and all conections are from the IP of the VM so i think the problem are the public accesibility of the VM but i don't know how to solved it...
If i activate my firewall rules to only allow traffic from LB and Cloudflare the web refuses all conections..
Any idea what i can do?
Thanks.
Cloud Support here, unfortunately, we do not have visibility into what is installed on your instance or what software caused the issue.
Generally speaking you're responsible for investigating the source of the vulnerability and taking steps to mitigate it.
I'm writing here some hints that will help you:
Make sure you keep your firewall rules in a sensible manner, e.g. is not a good practice to have a firewall rule to allow all ingress connections on port 22 from all source IPs for obvious reasons.
Since you've already been rooted, change all your passwords: within the Cloud SQL instance, within the GCE instance, even within the GCP project.
It's also a good idea to check who has access to your service accounts, just in case people that aren't currently working for you or your company still have access to them.
If you're using certificates revoke them, generate new ones and share them in a secure way and with the minimum required number of users.
Securing GCE instances is a shared responsability, in general, OWASP hardening guides are really good.
I'm quoting some info here from another StackOverflow thread that might be useful in your case:
General security advice for Google Cloud Platform instances:
Set user permissions at project level.
Connect securely to your instance.
Ensure the project firewall is not open to everyone on the internet.
Use a strong password and store passwords securely.
Ensure that all software is up to date.
Monitor project usage closely via the monitoring API to identify abnormal project usage.
To diagnose trouble with GCE instances, serial port output from the instance can be useful.
You can check the serial port output by clicking on the instance name
and then on "Serial port 1 (console)". Note that this logs are wipped
when instances are shutdown & rebooted, and the log is not visible
when the instance is not started.
Stackdriver monitoring is also helpful to provide an audit trail to
diagnose problems.
You can use the Stackdriver Monitoring Console to set up alerting policies matching given conditions (under which a service is considered unhealthy) that can be set up to trigger email/SMS notifications.
This quickstart for Google Compute Engine instances can be completed in ~10 minutes and shows the convenience of monitoring instances.
Here are some hints you can check on keeping GCP projects secure.
Hi my goal is to create Active Directory in AWS. I used simple AD and used 2 public and 2 private subnets within the same VPC with the private ones being for the domain controllers. I created an EC2 instance within the same VPC with Windows Server so that I can manage the AD. My EC2 instance joins the domain with no problem. My problem however is I cannot get my local machines on my network to join the AD, as the DC's, are of course private IP's and I cant change the DNS on my machine to these IP's unless on the same network.
Im guessing I need a VPN to join my local network to the Network in the AWS cloud.
Is there a way to achieve having AD in AWS without a VPN such as using an elastic IP with NAT to communicate to the DC's? Or maybe even promoting my EC2 instance to a DC then connecting the local machines DNS to the EC2 instances elastic IP?
Any help is much appreciated and let me know if I am missing any information or not explaining the goal clear enough.
Your question mentions Simple AD. My comments will be for Active Directory in AWS.
Setting up Active Directory in AWS and on-premises is not as easy as I would like it to be. This topic can fill a small book or as Amazon does it, multiple hour long videos. Watch a few while thinking up your solution.
1) Simple AD is not real Active Directory. It is Samba 4, which is very good, but is an Active Directory clone.
2) Do not, and I repeat do not, think about putting Active Directory on a public IP address to serve your on-premises users. The number of ports that you need to open and the risk is just not worth it.
3) Most, if not all, real solutions for configuring Active Directory on-premises and in AWS involve VPNs. Either Direct Connect (DX), hardware routers (Cisco) or site to site VPNs built from OpenSwan or Windows Server.
Note: OpenSwan is very easy to setup, so this is the route I would recommend if cost is a factor. Otherwise look at Cisco ASA type routers (lots of vendors here) for your office and setup a VPN with IPSEC. If cost is not a factor, absolutely go with Direct Connect (DX).
Note: I also use OpenVPN to connect to AD in AWS from home. This setup routes my workstation to a VPC in AWS and is so easy to setup and use. You could start with this to get comfortable with networking to a VPC. There are preconfigured OpenVPN setups in AWS marketplace that are free (user limited).
We're considering to implement an ELB in our production Amazon environment. It seems it will require that production server instances be synched by a nightly script. Also, there is a Solr search engine which will need to replicated and maintained for each paired server. There's also the issue of debugging - which server is it going to? If there's a crash, do you have to search both logs? If a production app isn't behaving, how do you isolate which one is is, or do you just deploy debugging code to both instances?
We aren't having issues with response time or server load. This seems like added complexity in exchange for a limited upside. It seems like it may be overkill to me. Thoughts?
You're enumerating the problems that arise when you need high availability :)
You need to consider how critical is the availability of the service and take that into account when defining what is the right solution or just over-engineering :)
Solutions to some caveats:
To avoid nightly syncs: Use an EC2 with NFS server and mount share in both EC2 instances. (Or use Amazon EFS when it's available)
Debugging problem: You can configure the EC2 instances behind the ELB to have public IPs, limited in the Security Groups just to the PCs of the developers, and when debugging point your /etc/hosts (or Windows equivalent) to one particular server.
Logs: store the logs in S3 (or in the NFS server commented above)
I need to serve a live stream to more than 10K users. Checking Adobe website it says that one EC2 instance of type m2.2xlarge is able to serve to just 10K users so I've some questions:
Does CloudFront allow to more users to connect than those 10K users allowed by the EC2 instance acting as a multiplexor of the original tream?
And based on the response of the above question:
If CloudFront allow more users to connect then, why should anyone need one m2.2xlarge EC2 instance if one with lower specs could do the same job and let CloudFront to multiplex the live stream?
If CludFront doesn't allow more users to connect than those 10K, what kind of architecture do I need? CloudFront + ELB + 2 or more EC2 instances with AMS installed a nd connected to another small EC2 instance with AMS installed which gets the stream from the live event?
CloudFront acts as a caching layer, for each edge location. If the content is not available at the edge location, it connects to EC2, retrieves the data and passes it on. So as far as I know, if using CloudFront, you shouldn't need such a large EC2 instance.
I've tested this extensively with static resources, I didn't need it for live streaming yet, but the same principles should apply.
This post on the AWS website from 2012 seems to confirm my hypothesis: http://aws.amazon.com/about-aws/whats-new/2012/03/29/amazon-cloudfront-improves-live-streaming-support-with-adobe-fms/
So basically, as long as the EC2 instance is strong enough to stream to all CloudFront edge location simultaneously, you should be fine.
I am planning to run a web-application and expecting a traffic of around 100 to 200 users.
Currently I have set up single Small instance on Amazon. This instance consist of everything – the Webserver(Apache) , the Database Server(MySQL) and the IMAP server( Dovcot). I am thinking of moving out my database server out of this instance and create a separate instance for it. Now my question is –
Do I get latency while communication between my webserver and Database server( Both hosted on separate instances on Amazon )
If yes, what is the standard way to overcome this ? ( or Do I need to set up a Virtual Private Cloud ?)
If you want your architecture to scale you should separate your web server from your database server.
The low latency that you will pay (~1-2ms even between multiple availability zone), will give you better performance as you can scale each tier separately.
You can add small (even micro) instances to handle more web requests behind a load balancer, without the need to duplicate an instance that has to have a database as well
You can add auto-scale group for your web server that will automatically scale your web server tier, based on usage load
You can scale up your DB instance, to have more memory, getting a better cache hit
You can add Elastic Cache between your web server and your database
You can use Amazon RDS as a managed database service, which remove the need for an instance for the database at all (you will pay only for the actual usage of the database in RDS)
Another benefit that you can have is better security on your database. If your database is on a separate instance, you can prevent access to it from the internet. You can use a security group that allows only sql connection from your web server and not http connection from the internet.
This configuration can run in a regular EC2 environment, without the usage of VPC. You can certainly add VPC for an even more control environment, without additional cost nor much increased complexity.
In short, for scalability and high availability you should separate your tiers (web and DB). You will probably also find yourself saving on cost as well.
Of course there will be latency when communicating between separate machines. If they are both in the same availability zone it will be extremely low, typically what you'd expect for two servers on the same LAN.
If they are in different availability zones in the same region, expect a latency on the order of 2-3ms (per information provided at the 2012 AWS re:Invent conference). That's still quite low.
Using a VPC will not affect latency. That does not give you different physical connections between instances, just virtual isolation.
Finally, consider using Amazon's RDB (Relational Database Service) instead of a dedicated EC2 instance for your MySql database. The cost is about the same, and Amazon takes care of the housekeeping.
Do I get latency while communication between my webserver and Database server( Both hosted on separate instances on Amazon )
Yes, but it's rather insignificant compared to the benefits gained by separating the roles.
If yes, what is the standard way to overcome this ? ( or Do I need to set up a Virtual Private Cloud ?)
VPC increases security and ease of management of the resources, it does not affect performance. A latency of a millisecond or two isn't normally problematic for a SQL database. Writes are transactional so data isn't accessible to other requests until it's 100% completed and committed. I/O throughput and availability are much more of a concern, which is why separating the database and the application is important.
I'd highly recommend that you take a look at RDS, which is AWS's version of a managed MySQL, Oracle, or MS SQL Server database server. This will allow you to easily setup and manage your database, including cross-availability zone replication and automated backups. I also wrote a blog post yesterday that's fairly relevant to your question.