detect www. using sed - amazon-web-services

I am new to sed and almost confused.
Here is what I have in the nginx folder of my project:
files:
"/tmp/45_nginx_https_rw.sh":
owner: root
group: root
mode: "000644"
content: |
#! /bin/bash
CONFIGURED=`grep -c "return 301 https" /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf`
if [ $CONFIGURED = 0 ]
then
sed -i '/listen 8080;/a \ if ($http_x_forwarded_proto = "http") { return 301 https://$host$request_uri; }\n' /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
logger -t nginx_rw "https rewrite rules added"
exit 0
else
logger -t nginx_rw "https rewrite rules already set"
exit 0
fi
The above code works like a charm and basically redirects all the request to https if they are http request.
However I need to add a piece to check if the url has www and redirect to non www.
so for example www.test.com will be redirected to test.com.
How can I achieve this?

The AWS recommended solution was to completely rewrite "/etc/nginx/sites-available/elasticbeanstalk-nginx-docker-proxy.conf"
You can do this by adding this file in .ebextensions: https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/security-configuration/https-redirect/docker-sc/https-redirect-docker-sc.config
I made one extra tweak for www redirect:
location / {
set $redirect 0;
if ($http_x_forwarded_proto != "https") {
set $redirect 1;
}
if ($host ~ ^www\.(?<domain>.+)$) {
set $redirect 1;
}
if ($http_user_agent ~* "ELB-HealthChecker") {
set $redirect 0;
}
if ($redirect = 1) {
return 301 https://ADDYOURDOMAINREDIRECTHERE$request_uri;
}
*One more note, your ec2 instance may already have a bunch of config files that you tried while finding a solution. I suggest deploying your app with NO config in .ebextensions, and then Rebuild Environment from the ELB console. This should give you a blank slate. Then simply add the file above to .ebextensions and redeploy.
*another note, you can ssh into your ec2 instance and verify that the file looks correct

Related

ingress-nginx host redirect in server snippet doesnt work

I'm using wildcard subdomain in the ingress and need to redirect any request that contains subdomain with www to non-www i.e.:
www.randomsubdomain.domain.com to randomsubdomain.domain.com
For this reason I tried to add server snippet to the ingress but it has no effect at all, I have tested my regular expression and it fits my problem but seem to be ignored by my ingress:
nginx.ingress.kubernetes.io/server-snippet: |
if ($host ~* "^(www\.)([0-9a-z]+\.domain\.com)$") {
return 301 https://$2$request_uri;
}
To achieve that you just need to add:
annotations:
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"

Errors adding Environment Variables to NodeJS Elastic Beanstalk

My configuration worked up until yesterday. I have added the nginx NodeJS https redirect extension from AWS. Now, when I try to add a new Environment Variable through the Elastic Beanstalk configuration, I get this error:
[Instance: i-0364b59cca36774a0] Command failed on instance. Return code: 137 Output: + rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf + service nginx stop Stopping nginx: /sbin/service: line 66: 27395 Killed env -i PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OPTIONS}. Hook /opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
When I look at the eb-activity.log, I see this error:
[2018-02-18T17:24:58.762Z] INFO [13848] - [Configuration update 1.0.61#112/ConfigDeployStage1/ConfigDeployPostHook/99_kill_default_nginx.sh] : Starting activity...
[2018-02-18T17:24:58.939Z] INFO [13848] - [Configuration update 1.0.61#112/ConfigDeployStage1/ConfigDeployPostHook/99_kill_default_nginx.sh] : Activity execution failed, because: + rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
+ service nginx stop
Stopping nginx: /sbin/service: line 66: 14258 Killed env -i PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OPTIONS} (ElasticBeanstalk::ExternalInvocationError)
caused by: + rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
+ service nginx stop
Stopping nginx: /sbin/service: line 66: 14258 Killed env -i PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OPTIONS} (Executor::NonZeroExitStatus)
What am I doing wrong? And what has changed recently since this worked fine when I changed an Environment Variable a couple months ago.
I had this problem as well and Amazon acknowledged the error in the documentation. This is a working restart script that you can use in your .ebextensions config file.
/opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash -xe
rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
status=`/sbin/status nginx`
if [[ $status = *"start/running"* ]]; then
echo "stopping nginx..."
stop nginx
echo "starting nginx..."
start nginx
else
echo "nginx is not running... starting it..."
start nginx
fi
service nginx stop exits with status 137 (Killed).
Your script starts with: #!/bin/bash -xe
The parameter -e makes the script exit immediately whenever something exits with a non-zero status.
If you want to continue the execution, you need to catch the exit status (137).
/opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash -xe
rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
status=`/sbin/status nginx`
if [[ $status = *"start/running"* ]]; then
set +e
service nginx stop
exitStatus = $?
if [ $exitStatus -ne 0 ] && [ $exitStatus -ne 137 ]
then
exit $exitStatus
fi
set -e
fi
service nginx start
The order of events looks like this to me:
Create a post-deploy hook to delete /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
Run a container command to delete /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
Run the post-deploy hook, which tries to delete /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
So it doesn't seem surprising to me that the post-deploy script fails as the file you are trying to delete probably doesn't exist.
I would try one of two things:
Move the deletion of the temporary conf file from the container command to the 99_kill_default_nginx.sh script, then remove the whole container command section.
Remove the line rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf from the 99_kill_default_nginx.sh script.
/sbin/status nginx seems not to work anymore. I updated the script to use service nginx status:
/opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash -xe
rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
status=$(service nginx status)
if [[ "$status" =~ "running" ]]; then
echo "stopping nginx..."
stop nginx
echo "starting nginx..."
start nginx
else
echo "nginx is not running... starting it..."
start nginx
fi
And the faulty script is STILL in amazon's docs... I wonder when they are going to fix it. It's been enough time already

Enabling CORS on Google App Engine for a Django Application

I have been trying to enable CORS headers on Google app engine but none of the methods that I found over the internet worked for me.
My application is on Python/Django and I want my frontend application (which is hosted separately) to be able to make API calls to my backend platform on Google App Engine.
The January 2017 release notes say that
We are changing the behavior of the Extensible Service Proxy (ESP) to deny cross-origin resource sharing (CORS) requests by default
It can be seenhere
And the solution to enable CORS given by them is to add the following snippet to the service's OpenAPI configuration.
"host": "echo-api.endpoints.YOUR_PROJECT_ID.cloud.goog",
"x-google-endpoints": [
{
"name": "echo-api.endpoints.YOUR_PROJECT_ID.cloud.goog",
"allowCors": "true"
}
],
...
So I followed this example and created two files in my code base
openapi.yml :
swagger: "2.0"
info:
description: "Google Cloud Endpoints APIs"
title: "APIs"
version: "1.0.0"
host: "echo-api.endpoints.<PROJECT-ID>.cloud.goog"
x-google-endpoints:
- name: "echo-api.endpoints.<PROJECT-ID>.cloud.goog"
allowCors: "true"
paths:
"/api/v1/sign-up":
post:
description: "Sends an email for verfication"
operationId: "signup"
produces:
- "application/json"
responses:
200:
description: "OK"
parameters:
- description: "Email address of the user"
in: body
name: email
required: true
schema:
type: string
- description: "password1"
in: body
name: password1
required: true
schema:
type: string
- description: "password2"
in: body
name: password2
required: true
schema:
type: string
openapi-appengine.yml:
swagger: "2.0"
info:
description: "Google Cloud Endpoints API fo localinsights backend server"
title: "Localinsights APIs"
version: "1.0.0"
host: "<PROJECT-ID>.appspot.com"
Then I ran this command:
gcloud service-management deploy openapi.yml
Then I edited my app.yml file to make it look like this (The addition was endpoints_api_service. Before adding this, the app was getting deployed without any errors):
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT myapp.wsgi
beta_settings:
cloud_sql_instances: <cloud instance>
runtime_config:
python_version: 3
automatic_scaling:
min_num_instances: 1
max_num_instances: 1
resources:
cpu: 1
memory_gb: 0.90
disk_size_gb: 10
env_variables:
DJANGO_SETTINGS_MODULE: myapp.settings.staging
DATABASE_URL: <dj-database-url>
endpoints_api_service:
name: "<PROJECT-ID>.appspot.com"
config_id: "<CONFIG-ID>"
Then I just deployed the application with
gcloud app deploy
Now, the app got deployed successfully but it is behaving strangely. All the requests which are supposed to return a 200 response still throw CORS error but the ones which return a 400 status do work.
For example - The sign up API expects these fields - email, password1, password2 where password1 should be same as password2. Now when I send correct parameters, I get HTTP 502 saying
No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin {origin-url} is therefore not allowed access. The response had HTTP status code 502
But when I send password1 not same as password2, I get HTTP 400 response which I am sure is coming from my code because the response is a dictionary written in the code if password1 and password2 do not match. Also in this case, the headers have Access-Control-Allow-Origin as * but in the former case, that was not true
I also checked my nginx error logs and it says
*27462 upstream prematurely closed connection while reading response header
What am I doing wrong here?
Is this the right way to enable CORS in GAE?
After banging my head for several days, I was able to figure out the the real problem. My database server was denying any connection to the webapp server.
Since in case of a HTTP 200 response, the webapp is supposed to make a database call, the webapp was trying to connect to the database server. This connection was taking too long and as soon as it reached beyond the NGINX's timeout time, NGINX used to send a response to the web browser with the status code as 502.
Since the 'access-control-allow-origin' header was being set from the webapp, NGINX did not set that header in its response. Hence the browser was interpreting it as a CORS denial.
As soon as I whitelisted my webapp's instance's IP address for the database server, things started running smoothly
Summary:
There is no need of openapi.yml file to enable CORS for a Django application on GAE flexible environment
Do not miss to check the NGINX logs :p
Update:
Just wanted to update my answer to specify the way through which you won't have to add you instance's IP to the whitelisted IP(s) of the SQL instance
Configure the DATABASES like this:
DATABASES = {
'HOST': <your-cloudsql-connection-string>, # This is the tricky part
'ENGINE': <db-backend>,
'NAME': <db-name>,
'USER': <user>,
'PASSWORD': <password>
}
Note the HOST key in the databases. GAE has a way through which you won't have to whitelist your instance's IP but for that to work, the host should be the cloudsql-connection-string and NOT the IP of the SQL instance.
If you are not sure what's your cloudsql-connection-string, go to the Google cloud platform dashboard and select the SQL tab under the Storage section. You should see a table with a column Instance connection name. The value under this column is your cloudsql-connection-string.
Nginx as your reverse proxy, so, as the gateway to your server, should be who manage CORS against client browser requests, as first contact from beyond to your system. Should not be any of the backend servers (neither your database, neither anything).
Here you got my default config to enable CORS in nginx from Ajax calls to a REST service of my own (backserver glassfish). Feel free to check and use it and hope it serves to you.
server {
listen 80; ## listen for ipv4; this line is default and implied
server_name codevault;
#Glassfish
location /GameFactoryService/ {
index index.html;
add_header Access-Control-Allow-Origin $http_origin;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-NginX-Proxy true;
proxy_ssl_session_reuse off;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:18000/GameFactoryService/;
}
#static content
location / {
root /usr/share/nginx_static_content;
}
error_page 500 501 502 503 504 505 506 507 508 509 510 511 /50x.html;
#error
location = /50x.html {
add_header Access-Control-Allow-Origin $http_origin;
internal;
}
}

.ebextensions not executing not uploading and creating files

I am trying to follow these instructions to force SSL on AWS Beanstalk.
I believe this is the important part.
files:
"/tmp/45_nginx_https_rw.sh":
owner: root
group: root
mode: "000644"
content: |
#! /bin/bash
CONFIGURED=`grep -c "return 301 https" /opt/elasticbeanstalk/support/conf/webapp_healthd.conf`
if [ $CONFIGURED = 0 ]
then
sed -i '/listen 80;/a \ if ($http_x_forwarded_proto = "http") { return 301 https://$host$request_uri; }\n' /opt/elasticbeanstalk/support/conf/webapp_healthd.conf
logger -t nginx_rw "https rewrite rules added"
exit 0
else
logger -t nginx_rw "https rewrite rules already set"
exit 0
fi
container_commands:
00_appdeploy_rewrite_hook:
command: cp -v /tmp/45_nginx_https_rw.sh /opt/elasticbeanstalk/hooks/appdeploy/enact
01_configdeploy_rewrite_hook:
command: cp -v /tmp/45_nginx_https_rw.sh /opt/elasticbeanstalk/hooks/configdeploy/enact
02_rewrite_hook_perms:
command: chmod 755 /opt/elasticbeanstalk/hooks/appdeploy/enact/45_nginx_https_rw.sh /opt/elasticbeanstalk/hooks/configdeploy/enact/45_nginx_https_rw.sh
03_rewrite_hook_ownership:
command: chown root:users /opt/elasticbeanstalk/hooks/appdeploy/enact/45_nginx_https_rw.sh /opt/elasticbeanstalk/hooks/configdeploy/enact/45_nginx_https_rw.sh
For some reason the file is not being uploaded or created.
I also tried this a sudo command for the container commands starting with 00 and 01.
I also manually ssh into the server, manually created the file. Then locally used the aws elasticbeanstalk restart-app-server --environment-name command to restart the server. And this still did not work.
Any help would be greatly appreciated.

Parsing .ssh/config for proxy information

I would like to parse the .ssh/config file for proxy information contained therein and display each host that has related proxy information with the proxy information. Hosts that do not have any proxy information should be filtered out. Man page for .ssh/config: http://man-wiki.net/index.php/5:ssh_config
This should be done from a Unix shell script under Bash, so a standard tool like Perl, awk or sed is preferred.
Example input file:
Host ssh.foo.com
User ssh
HostName ssh.foo.com
Port 443
ProxyCommand /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
Host ci
HostName 127.0.0.2
User ci
Host nightly
HostName 192.168.1.1
User goodnight
Host foobar.org
User git
HostName foobar.org
Port 443
ProxyCommand /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
Host integration
HostName 192.168.1.2
User int
The expected output should look like this:
Host: ssh.foo.com - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
Host: foobar.org - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
The difficulty here is that the search has to cover multiple lines.
Try following awk command:
awk '
$1 == "Host" {
host = $1 ": " $2;
next;
}
$1 == "ProxyCommand" {
$1 = "";
sub( /^[[:space:]]*/, "" );
printf "%s - Proxy: %s\n", host, $0;
}
' .ssh/config
It yields:
Host: ssh.foo.com - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
Host: foobar.org - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
an awk oneliner may work for your requirement:
awk -v RS="" '/Proxy/{gsub(/\n/,"");gsub(/\s*User.*ProxyCommand/,"- Proxy:");print}'file
test (a.txt is your input file)
kent$ awk -v RS="" '/Proxy/{gsub(/\n/,"");gsub(/\s*User.*ProxyCommand/,"- Proxy:");print}' a.txt
Host ssh.foo.com - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
Host foobar.org - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
Just for kicks, here is a nasty sed command that seems to accomplish this.
sed -n '/^[Hh]ost/{:r;h;:l;n;/^[Hh]ost/!{/[Pp]roxy/H;bl};/^[Hh]ost/{x;/[Pp]roxy/{s/\n\s*/ - /;s/[Cc]ommand/:/;s/[Hh]ost/\0:/;p};x;br}}' ~/.ssh/config
This is more for demonstration purposes anyway since the other solutions are much easier to grasp. A commented view of this:
/^[Hh]ost/ { # Begin on a Host line
:restart # Label to jump back to
h # Copy the pattern space to the save space
:loop # Label to jump to when we don't want to wipe out the save space
n # Read in the next line
/^[Hh]ost/!{ # For lines that aren't host definitions
/[Pp]roxy/H # If they are proxy configurations, append the line to the save space
b loop # Go to :loop
}
/^[Hh]ost/{ # If they are *not* host definitions
x # Exchange the pattern and save spaces
/[Pp]roxy/{ # If the new pattern space contains a proxy configuration
s/\n\s*/ - / # Insert the required separator
s/[Cc]ommand/:/ # Change "ProxyCommand" to "Proxy:"
s/[Hh]ost/\0:/ # Change "Host" to "Host:"
p # Print the pattern space
}
x # Exchange the pattern and save spaces
b restart # Go to :restart
}
}
Example output:
Host: ssh.foo.com - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
Host: foobar.org - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
Have you checked out the Net::SSH::Perl::Config module ? It claims to read configs in the standard ssh format - albeit with a subset of keywords.
The configuration files should be in the same format used for the ssh
command line program; see the ssh manpage for information on this
format. Net::SSH::Perl::Config understands a subset of the
configuration directives that can live in these files; this subset
matches up with the functionality that Net::SSH::Perl can support.
Unknown keywords will simply be skipped.
One way using GNU sed:
sed -nre '/^Host/h;/ProxyCommand/{H;x;s/(Host)(.*)\n +[^ ]+ /\1:\2 - Proxy: /p}' file.txt
Results:
Host: ssh.foo.com - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
Host: foobar.org - Proxy: /usr/local/bin/corkscrew proxy 8080 %h %p ~/.ssh/proxyauth
try this:
#! /usr/bin/perl
use strict;
use warnings;
open FD, "DATA" || die $!;
my $line = do {local $/; <FD>};
my #a = split /Host\b/, $line;
foreach (#a) {
if (defined $_ and /^\s*(\S+).*ProxyCommand\s+([^\n]+)/s) {
print "$1 - Proxy: $2\n";
}
}