I got a new laptop and planning to dedicate the current laptop as a central log monitoring system for the server clusters already set up on AWS. AWS servers have static IP, while my personal laptop will be connected to Wifi. The clusters receive low to moderate traffic and there aren't many logs generated.
To use the laptop as a central log monitoring system, I can do one of these things:
Stream logs in realtime(Using streams to reduce reconnection overheads)
HTTP Long Polling(Can't push as my ISP doesn't allow me a static IP)
Make a VPN server and figure out some way to push/poll logs.
I think the 1st option(streaming logs) looks the most promising.
Is there some better way to this?
Also, how do I stream logs in this setup considering clients have static IP while my central server has dynamic IP?
Are there any open-source/existing services that achieves this already(Why re-invent the wheel when you have a start!)?
Thank you in advance!
Related
I have made a web application (Play Framework, Cassandra) which I'll not put in production on GCP . However, I am not well versed with networking and systems administration. The application would be containerised and I'll use K8s to create a cluster containing 2 pods for Play web application and 3 pods for Cassandra (for replication). There also be a load balancer service in front of the play application.
I suppose the above configuration is still vulnerable to DDOS attack. How can I prevent it on GCP?
The communication between the browser and server is not encrypted (eg. the passwords is being sent in plain text. Could could I enable encryption on GCP?
Any other tips on creating a reliable production system would be much appreciated. So far I have only worked on my laptop
Please find the below responses
I suppose the above configuration is still vulnerable to DDOS
attack. How can I prevent it on GCP?
https://cloud.google.com/files/GCPDDoSprotection-04122016.pdf, this can help you with DDOS protection. If you are using APIGEE, then APIGEE edge can also help you
The communication between the browser and server is not encrypted
(eg. the passwords is being sent in plain text. Could could I enable
encryption on GCP?
you will have to install certificated in general and ensure that the data is sent over a https network. You can also try sslforfree
Any other tips on creating a reliable production system would be
much appreciated. So far I have only worked on my laptop
since you are using K8s to run cassandra please ensure you are using some sort of volumes to store the data.
I would like to know what is the best and the easiest solution
to protect http server deployed on AWS cloud against DOS attacks
I know that there is AWS Advanced Shield
that can be turned on for that purpose
however it is too expensive (3000$ per month)
https://aws.amazon.com/shield/pricing/
System architecture
HTTP request -> Application Load Balancer -> EC2
Nginx server is installed on this machine
Nginx server is configured with rate limiting
Nginx server responds with 429 code when too many requests are send from one IP
Nginx server is generating log files (access.log, error.log)
AmazonCloudWatchAgent is installed on this machine
AmazonCloudWatchAgent listen on log files
AmazonCloudWatchAgent send changes from log files to specific CloudWatch Log groups
Logs from all EC2 machines are centralized in on place (CloudWatch Log groups)
I can configure CloudWatch Logs Metric Filters
to send me alarms when too many 429 requests happen from one IP number
In that way I can manually block particular IP in Network ACL
and cut off all requests from bad IP number in a lower network layer
and protect my AWS resources from being drained
I would like to do it somehow automatically
What is the easiest and the cleanest way to do it?
Note that, per the AWS Shield pricing documentation:
AWS Shield Standard provides protection for all AWS customers from
common, most frequently occurring network and transport layer DDoS
attacks that target your web site or application at no additional
charge.
For a more comprehensive discussion on DDoS mitigation, see:
Denial of Service Attack Mitigation on AWS
AWS Best Practices for DDoS Resiliency
There is no one straightforward way to block DDOS to your infrastructure. However, there are a few techniques and best practices which you can follow to at least protect the infrastructure. DDOS attacks can be stopped by analyzing and patching it at the same moment.
You may consider using external services listed below to block ddos at some extent:
Cloudflare: https://www.cloudflare.com/en-in/ddos/
Imperva Incapsula:
https://www.imperva.com/products/ddos-protection-services/
I have tried both in the production system and they are pretty decent. Cloudflare is right now handling 10% of total internet traffic, they know about the good and bad traffic.
They are not much expensive comparative to shield. You may integrate it with your infrastructure as a code in order to automate for all of your services.
Disclaimer: I am not associated in any way with any of the services I recommended above.
i am traying to figure it out how is the best way to deploy a TCP/IP and UDP service on Amazon AWS.
I made a previous research to my question and i can not find anything. I found others protocols like HTTP, MQTT but no TCP or UDP
I need to refactor a GPS Tracking service running right now in AMAZON EC2. The GPS devices sent the position data using udp and tcp protocol. Every time a message is received the server have to respond with an ACKNOWLEDGE message, giving the reception confirmation to the gps device.
The problem i am facing right now and is the motivation to refactor is:
When the traffic increase, the server is not able to catch up all the messages.
I try to solve this issue with load balancer and autoscaling but UDP is not supported.
I was wondering if there is something like Api Gateway, which gave me a tcp or udp endpoint, leave the message on a SQS queue and process with a lambda function.
Thanks in advance!
Your question really doesn't make a lot of sense - you are asking how to run a service without running a server.
If you have reached the limits of a single instance, and you need to grow, look at using the AWS Network Load Balancer with an autoscaled group of EC2 instances. However, this will not support UDP - if you really need that, then you may have to look at 3rd party support in the AWS Marketplace.
Edit: Serverless architectures are designed for http based application, where you send a request and get a response. Since your app is TCP based, and uses persistent connections, most existing serverless implementations simply won't support it. You will need to rewrite your app to support http, or use traditional server based infrastructures that can support persistent connections.
Edit #2: As of Dec. 2018, API gateway supports WebSockets. This probably doesn't help with the original question, but opens up other alternatives if you need to run lambda code behind a long running connection.
If you want to go more Serverless, I think the ECS Container Service has instances that accept TCP and UDP. Also take a look at running Docker Containers with with Kubernetes. I am not sure if they support those protocols, but I believe they do.
If not, some EC2 instances with load balancing can be your best bet.
I want to run a dpdk experiment using Amazon EC2 service. But there are a great number of services in AWS. I don't know which one to choose.
My experiment need two servers connected together using 10Gbps network adpater supporting dpdk. I run pktgen-dpdk on one server to send packets towards the other server. And another dpdk application will run in the other server to deal with these packets.
I think I can rent servers such c4.8xlarge c4.4xlarge. But I don't know how to set up the local network between them. The local network should have low latency.
Any suggestions will be appreciated! Thank you!
You're looking for Virtual Private Cloud (VPC). An AWS EC2 "instance" like your c4.8xlarge is just a machine. The VPC and several other components allow you to set up a broader network, routing, security groups (basically, a firewall) and other networking capabilities, including in your case a Gateway, which would allow your dpkg system to look out onto the Internet to find dependencies.
The in-network latency is extremely low, < 1ms in our experience. I think most current EC2 instances support 10Gbps networking and other speedy network capabilities.
Good Day,
I have been using AWS quite a bit for my cloud based system for a hardware project. Using SimpleDB and the notification service provided is great.
However, I need a backend on AWS that basically listens to requests coming in, processes it and sends it back to a particular address. Some kind of UDP service.
I could easily write a c#/c++ app for it, but i am not sure if I can host it on AWS. Does anyone know how this works?
Short answer: yes.
EC2 instances are just like any other virtual machine, obviously you can put in a server that listens to UDP. Configuring the network for this is, of course, slightly more complicated, but possible. The one thing making it more complicated is that with UDP you will not be able to enjoy the load balancer service that Amazon offers, as it (currently) only supports TCP-based protocols.
So, if you have one server you wish to put on the internet, the procedure is probably same as what you'd do with a TCP server: set up a server and an elastic IP pointing to it, and then have your clients connect to it (by knowing the elastic IP you've been allocated, or by referring to that IP via a DNS resolution). If you have multiple servers you wish to set up, answering the same address, life is a bit more complicated. With TCP, you could have set up an Amazon load balancer and assign your elastic IP to the load balancer. If you'd want a load balancer for UDP, the Amazon stock load balancer can't do that, but you can still find a software load balancer (there are hundreds of them on Amazon's public images library) to set up.
Nginix has an Amazon image that will load balance UDP for $2,500/yr or you can launch your own EC2 instance and use open source Nginx.
My specific use case was for a UDP logging service, if you can use hostnames Route 53 could be a scalable managed solution as well.