Trying to clean up my ElasticSearch tracer output - regex

I use elasticsearch a lot and I needed a way to see the request to elasticsearch server.
So I found this bash script:
sudo tcpdump -A -nn -s 0 'tcp dst port 9200 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -i lo
Works well, the problem is it outputs a lot of junk:
11:40:49.690863 IP 127.0.0.1.50270 > 127.0.0.1.9200: Flags [P.], seq 2594084191:2594084256, ack 2674569216, win 342, options [nop,nop,TS val 910075 ecr 910075], length 65
E..uw.#.#............^#...._.j.....V.i.....
..HEAD / HTTP/1.1
Host: localhost:9200
Connection: keep-alive
11:40:49.707181 IP 127.0.0.1.50271 > 127.0.0.1.9200: Flags [P.], seq 3009491178:3009491248, ack 3451175088, win 342, options [nop,nop,TS val 910080 ecr 910079], length 70
E..z..#.#.jc........._#..a0........V.n.....
..GET /_nodes HTTP/1.1
Host: localhost:9200
Connection: keep-alive
11:40:52.215800 IP 127.0.0.1.50272 > 127.0.0.1.9200: Flags [P.], seq 148869443:148869508, ack 2833610757, win 342, options [nop,nop,TS val 910707 ecr 910707], length 65
E..ut3#.#..M.........`#....C..x....V.i.....
.sHEAD / HTTP/1.1
Host: localhost:9200
Connection: keep-alive
I just want to keep the JSON.
So I though about going by elimination.
I added this regex:
sudo tcpdump -A -nn -s 0 'tcp dst port 9200 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -i lo | grep -v "[0-9]{2}[:][0-9]{2}[:][0-9]{2}\.[0-9]*[ ][I][P][ ][0-9]{1,3}\.[0 -9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]*[ ][\>].*"
Here is the regex in action: https://regex101.com/r/aS6kS2/1
Anyways, what I am trying to accomplish here is to remove:
11:40:49.707181 IP 127.0.0.1.50271 > 127.0.0.1.9200: Flags [P.], seq 3009491178:3009491248, ack 3451175088, win 342, options [nop,nop,TS val 910080 ecr 910079], length 70
E..z..#.#.jc........._#..a0........V.n.....
..GET /_nodes HTTP/1.1
Host: localhost:9200
Connection: keep-alive
So that only the body/JSON is kept.
I was trying to go line by line but for now it's not even removing the first one that was the target of my regex.

This will do what you've told us about so far (remove the middle chunk of text from the sequence of 3 chunks of text):
$ awk -v RS= -v ORS='\n\n' 'NR!=2' file
11:40:49.690863 IP 127.0.0.1.50270 > 127.0.0.1.9200: Flags [P.], seq 2594084191:2594084256, ack 2674569216, win 342, options [nop,nop,TS val 910075 ecr 910075], length 65
E..uw.#.#............^#...._.j.....V.i.....
..HEAD / HTTP/1.1
Host: localhost:9200
Connection: keep-alive
11:40:52.215800 IP 127.0.0.1.50272 > 127.0.0.1.9200: Flags [P.], seq 148869443:148869508, ack 2833610757, win 342, options [nop,nop,TS val 910707 ecr 910707], length 65
E..ut3#.#..M.........`#....C..x....V.i.....
.sHEAD / HTTP/1.1
Host: localhost:9200
Connection: keep-alive
If that's not what you want then edit your question to clarify as I suggested in my comment.

Related

Django unable to send email using uWSGI

My django project sends email verification correctly till its using uWSGI, when i run it with uWSGI i am unable to send email, it sometimes sends as well sometimes it doesnt.
it doesnt show any error in console
Nginx config:
upstream django {
server unix://ucurs/ucurs.sock; # for a file socket
}
# configuration of the server
server {
# the port your site will be served on
listen 7000;
# the domain name it will serve for
server_name 127.0.0.1; # substitute your machine's IP address or FQDN
charset utf-8;
# max upload size
client_max_body_size 75M; # adjust to taste
location /log {
access_log off;
error_log off;
uwsgi_pass django;
include /ucurs/uwsgi_params;
}
# Django media
location /media {
alias /ucurs/media; # your Django project's media files - amend as required
}
location /static {
alias /ucurs/static; # your Django project's static files - amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass django;
include /ucurs/uwsgi_params; # the uwsgi_params file you installed
}
}
Django library used for sending email : https://pypi.org/project/django-email-verification/
Django email config
EMAIL_FROM_ADDRESS = 'noreply#ucurs.com'
EMAIL_MAIL_SUBJECT = 'Confirm your email'
EMAIL_MAIL_HTML = 'mail_body.html'
EMAIL_MAIL_PLAIN = 'mail_body.txt'
EMAIL_TOKEN_LIFE = 60 * 60
EMAIL_PAGE_TEMPLATE = 'confirm_template.html'
EMAIL_PAGE_DOMAIN = 'https://backend.ucurs.com/'
EMAIL_HOST = 'smtp.kryptohive.com'
EMAIL_PORT = 587
EMAIL_HOST_USER = 'noreply#ucurs.com'
EMAIL_HOST_PASSWORD = '******'
EMAIL_USE_TLS = True
uswgi.ini :
# mysite_uwsgi.ini file
[uwsgi]
# Django-related settings
# the base directory (full path)
chdir = .
# Django's wsgi file
module = ucurs_backend.wsgi
logto = report/uswgi_logs
# the virtualenv (full path)
# process-related settings
# master
master = true
# maximum number of worker processes
processes = 10
# the socket (use the full path to be safe
socket = ucurs.sock
# ... with appropriate permissions - may be needed
chown-socket = www-data:www-data
chmod-socket = 777
uid = www-data
gid = www-data
# clear environment on exit
vacuum = true
byte_size = 65535
uswgi_params :
uwsgi_param QUERY_STRING $query_string;
uwsgi_param REQUEST_METHOD $request_method;
uwsgi_param CONTENT_TYPE $content_type;
uwsgi_param CONTENT_LENGTH $content_length;
uwsgi_param REQUEST_URI $request_uri;
uwsgi_param PATH_INFO $document_uri;
uwsgi_param DOCUMENT_ROOT $document_root;
uwsgi_param SERVER_PROTOCOL $server_protocol;
uwsgi_param REQUEST_SCHEME $scheme;
uwsgi_param HTTPS $https if_not_empty;
uwsgi_param REMOTE_ADDR $remote_addr;
uwsgi_param REMOTE_PORT $remote_port;
uwsgi_param SERVER_PORT $server_port;
uwsgi_param SERVER_NAME $server_name;
EDIT 2
Tcpdump from django server shows no traffic for mail, so issue is in uswgi
09:20:35.298648 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [S], seq 1191300539, win 64240, options [mss 1460,sackOK,TS val 912819442 ecr 0,nop,wscale 7], length 0
09:20:35.298775 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [S.], seq 1210865976, ack 1191300540, win 65160, options [mss 1460,sackOK,TS val 3498762181 ecr 912819442,nop,wscale 7], length 0
09:20:35.298815 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [.], ack 1, win 502, options [nop,nop,TS val 912819442 ecr 3498762181], length 0
09:20:35.298895 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [P.], seq 1:9, ack 1, win 502, options [nop,nop,TS val 912819442 ecr 3498762181], length 8
09:20:35.298911 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [.], ack 9, win 509, options [nop,nop,TS val 3498762181 ecr 912819442], length 0
09:20:35.301343 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [P.], seq 1:2, ack 9, win 509, options [nop,nop,TS val 3498762183 ecr 912819442], length 1
09:20:35.301356 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [.], ack 2, win 502, options [nop,nop,TS val 912819444 ecr 3498762183], length 0
09:20:35.303706 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [P.], seq 2:15, ack 50, win 509, options [nop,nop,TS val 3498762186 ecr 912819444], length 13
09:20:35.303795 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [P.], seq 50:91, ack 15, win 502, options [nop,nop,TS val 912819447 ecr 3498762186], length 41
09:20:35.306641 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [P.], seq 15:341, ack 91, win 509, options [nop,nop,TS val 3498762189 ecr 912819447], length 326
09:20:35.307576 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [P.], seq 91:116, ack 341, win 501, options [nop,nop,TS val 912819451 ecr 3498762189], length 25
09:20:35.308579 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [P.], seq 341:374, ack 116, win 509, options [nop,nop,TS val 3498762191 ecr 912819451], length 33
09:20:35.308966 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [P.], seq 116:222, ack 374, win 501, options [nop,nop,TS val 912819452 ecr 3498762191], length 106
09:20:35.315080 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [P.], seq 374:421, ack 222, win 509, options [nop,nop,TS val 3498762197 ecr 912819452], length 47
09:20:35.319524 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [P.], seq 222:336, ack 421, win 501, options [nop,nop,TS val 912819463 ecr 3498762197], length 114
09:20:35.322154 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [P.], seq 421:468, ack 336, win 509, options [nop,nop,TS val 3498762204 ecr 912819463], length 47
09:20:35.362825 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [.], ack 468, win 501, options [nop,nop,TS val 912819506 ecr 3498762204], length 0
09:20:35.605078 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [P.], seq 336:873, ack 468, win 501, options [nop,nop,TS val 912819748 ecr 3498762204], length 537
09:20:35.614358 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [P.], seq 468:531, ack 873, win 505, options [nop,nop,TS val 3498762496 ecr 912819748], length 63
09:20:35.614387 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [.], ack 531, win 501, options [nop,nop,TS val 912819757 ecr 3498762496], length 0
09:20:35.616039 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [P.], seq 873:1076, ack 531, win 501, options [nop,nop,TS val 912819759 ecr 3498762496], length 203
09:20:35.619838 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [P.], seq 531:594, ack 1076, win 504, options [nop,nop,TS val 3498762502 ecr 912819759], length 63
09:20:35.620982 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [P.], seq 1076:1621, ack 594, win 501, options [nop,nop,TS val 912819764 ecr 3498762502], length 545
09:20:35.622083 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [P.], seq 594:614, ack 1621, win 501, options [nop,nop,TS val 3498762504 ecr 912819764], length 20
09:20:35.629989 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [P.], seq 1621:1626, ack 614, win 501, options [nop,nop,TS val 912819773 ecr 3498762504], length 5
09:20:35.633474 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [F.], seq 1626, ack 614, win 501, options [nop,nop,TS val 912819776 ecr 3498762504], length 0
09:20:35.633998 IP psql_db.prod_default.postgresql > ce2fc225de38.56560: Flags [F.], seq 614, ack 1627, win 501, options [nop,nop,TS val 3498762516 ecr 912819773], length 0
09:20:35.634021 IP ce2fc225de38.56560 > psql_db.prod_default.postgresql: Flags [.], ack 615, win 501, options [nop,nop,TS val 912819777 ecr 3498762516], length 0
09:20:38.494868 IP6 fe80::a8a0:99ff:fe25:6d1f > ip6-allrouters: ICMP6, router solicitation, length 16
09:20:38.496166 IP ce2fc225de38.35901 > one.one.one.one.domain: 58550+ PTR? f.1.d.6.5.2.e.f.f.f.9.9.0.a.8.a.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
09:20:38.497435 IP one.one.one.one.domain > ce2fc225de38.35901: 58550 NXDomain 0/0/0 (90)
09:20:38.497661 IP ce2fc225de38.48162 > dns.google.domain: 58550+ PTR? f.1.d.6.5.2.e.f.f.f.9.9.0.a.8.a.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
09:20:38.499486 IP dns.google.domain > ce2fc225de38.48162: 58550 NXDomain 0/1/0 (154)
09:20:38.500424 IP ce2fc225de38.37933 > one.one.one.one.domain: 58885+ PTR? 1.1.1.1.in-addr.arpa. (38)
09:20:38.501094 IP one.one.one.one.domain > ce2fc225de38.37933: 58885 1/0/0 PTR one.one.one.one. (67)
09:20:38.502256 IP ce2fc225de38.39070 > one.one.one.one.domain: 65529+ PTR? 8.8.8.8.in-addr.arpa. (38)
09:20:38.502846 IP one.one.one.one.domain > ce2fc225de38.39070: 65529 1/0/0 PTR dns.google. (62)
09:21:31.742925 IP6 fe80::a8a0:99ff:fe25:6d1f > ip6-allrouters: ICMP6, router solicitation, length 16
09:23:12.094915 IP6 fe80::a8a0:99ff:fe25:6d1f > ip6-allrouters: ICMP6, router solicitation, length 16
09:26:32.798927 IP6 fe80::a8a0:99ff:fe25:6d1f > ip6-allrouters: ICMP6, router solicitation, length 16
EDIT 3
I have finally confiremd issue is in uswgi configurations, as i tried proxy pass django development server and it works fine
proxy_pass http://ucurs_backend:8000;
# uwsgi_pass django;
# include /ucurs/uwsgi_params; # the uwsgi_params file you installed
Observation
closing uwsgi server sends all mails

Got 404 from http://169.254.170.2/$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

On AWS ECS or AWS CodeBuild etc, when trying to retrieve as credentials using:
http://169.254.170.2/$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
suddenly since Feb 7, 2019 - I got 404 not found !
curl -qL -o aws_credentials.json http://169.254.170.2/$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
The expected result should be a valid json of the AWS Credentials session
After short investigation:
I found that $AWS_CONTAINER_CREDENTIALS_RELATIVE_URI already starts by a slash '/'
[e.g AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/xxxx-xxxx-xxxx-xxxx-xxxxx]
Solution: just remove the slash after the IP.*
e.g http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
TL;DR;
I run curl with -v on AWS CodeBuild:
> GET //v2/credentials/xxxx-xxxx-xxxx-xxxx-xxxxx HTTP/1.1
> Host: 169.254.170.2
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
Conclusion: since Feb 6 or 7 2019, AWS add a strict check and broke the request with 404
for double slash //

AWS EC2 userdata WGET via link-local address times out

AWS EC2 instance uses the link-local address (169.254.169.254) in order to distribute user-data. So, I booted a Linux instance to experiment the same with some initial network setup scripts to get the data. Following is the setup before I requested to pull the information.
iptable:
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 06:43:0f:bf:cd:6c brd ff:ff:ff:ff:ff:ff
inet 169.254.169.15/16 brd 169.254.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::443:fff:febf:cd6c/64 scope link
valid_lft forever preferred_lft forever
route
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.169.15
Requested: wget http://169.254.169.254
REQUEST TIME OUT
tcpdump
00:00:00.000000 ARP, Request who-has 169.254.169.254 tell 169.254.169.15, length 28
00:00:00.000115 ARP, Reply 169.254.169.254 is-at <MAC REMOVED>, length 42
00:00:00.000119 IP 169.254.169.15.60712 > 169.254.169.254.80: Flags [S], seq 2929127779, win 29200, options [mss 1460,sackOK,TS val 2525905560 ecr 0,nop,wscale 7], length 0
00:00:01.048018 IP 169.254.169.15.60712 > 169.254.169.254.80: Flags [S], seq 2929127779, win 29200, options [mss 1460,sackOK,TS val 2525906608 ecr 0,nop,wscale 7], length 0
00:00:03.096020 IP 169.254.169.15.60712 > 169.254.169.254.80: Flags [S], seq 2929127779, win 29200, options [mss 1460,sackOK,TS val 2525908656 ecr 0,nop,wscale 7], length 0
00:00:07.128021 IP 169.254.169.15.60712 > 169.254.169.254.80: Flags [S], seq 2929127779, win 29200, options [mss 1460,sackOK,TS val 2525912688 ecr 0,nop,wscale 7], length 0
00:00:15.192051 IP 169.254.169.15.60712 > 169.254.169.254.80: Flags [S], seq 2929127779, win 29200, options [mss 1460,sackOK,TS val 2525920752 ecr 0,nop,wscale 7], length 0
00:00:31.576027 IP 169.254.169.15.60712 > 169.254.169.254.80: Flags [S], seq 2929127779, win 29200, options [mss 1460,sackOK,TS val 2525937136 ecr 0,nop,wscale 7], length 0
00:00:36.696009 ARP, Request who-has 169.254.169.254 tell 169.254.169.15, length 28
00:00:36.696189 ARP, Reply 169.254.169.254 is-at <MAC REMOVED>, length 42
00:01:03.832035 IP 169.254.169.15.60712 > 169.254.169.254.80: Flags [S], seq 2929127779, win 29200, options [mss 1460,sackOK,TS val 2525969392 ecr 0,nop,wscale 7], length 0
00:01:08.952026 ARP, Request who-has 169.254.169.254 tell 169.254.169.15, length 28
00:01:08.952182 ARP, Reply 169.254.169.254 is-at <MAC REMOVED>, length 42
Something is blocking Acknowledgement packets. What am I missing? Please advise!
If you need any other information, please let me know!

sf::Http::sendRequest never returns

I've written a simple web service using pistache. I'm seding requests to it using sf::Http and sf::Http::Request classes. However, call of sf::Http::sendRequest never returns, even though I specified a 250 ms timeout. The thing happens only with requests to my website. If I send GET request to www.google.com the method returns correct response quite quickly.
Here's the client-side code sample:
sf::Http http("http://192.168.1.10", 8080);
sf::Http::Request request("/highscores", sf::Http::Request::Method::Get);
request.setHttpVersion(1, 1);
//the call below never returns
auto response = http.sendRequest(request, sf::seconds(0.25f));
std::cout << response.getBody();
The service response seems correct in browser and in curl:
$ curl -v 192.168.1.10:8080/highscores
* Trying 192.168.1.10...
* Connected to 192.168.1.10 (192.168.1.10) port 8080 (#0)
> GET /highscores HTTP/1.1
> Host: 192.168.1.10:8080
> User-Agent: curl/7.47.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Connection: Keep-Alive
< Content-Length: 2
<
* Connection #0 to host 192.168.1.10 left intact
[]%
Using strace on my application shows that it sends correct request and even at some point it receives the correct response:
$ strace -s 192 ./sfmlApplication
...
sendto(20, "GET /highscores HTTP/1.1\r\nconnection: close\r\ncontent-length: 0\r\ncontent-type: application/json\r\nfrom: user#sfml-dev.org\r\nhost: 192.168.1.10\r\nuser-agent: libsfml-network/2.x\r\n\r\n", 176, MSG_NOSIGNAL, NULL, 0) = 176
recvfrom(20, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nConnection: Keep-Alive\r\nContent-Length: 2\r\n\r\n[]", 1024, MSG_NOSIGNAL, NULL, NULL) = 96
recvfrom(20,
These are the last lines from strace output, after recvfrom(20, the program stops responding and has to be killed.
And the top of stack trace of blocked operation is:
recv() at 0x7ffff7bcd10f
sf::TcpSocket::receive() at 0x7ffff77b12c0
sf::Http::sendRequest() at 0x7ffff77ad5ed
SFML Version: 2.3.2
System: Fedora 4.8.4-200.fc24.x86_64
Any ideas why the sf::Http::sendRequest method call never returns?

Extract client IP address from tcpdump (on OpenBSD)

I am trying to extract and collect some client IP addresses using the command tcpdump, but just can't figure out how to get grep/sed/awk to give me just the first IP address of the output. If I run the following
# tcpdump -t -i vlan4 -n ip dst host 192.168.2
tcpdump: listening on vlan4, link-type EN10MB
tcpdump: WARNING: compensating for unaligned libpcap packets
10.28.6.1.29447 > 192.168.2.62.22: P 3758838950:3758839034..
10.28.6.1.29447 > 192.168.2.62.22: P 84:136(52) ack 117..
10.28.6.1.29447 > 192.168.2.62.22: P 136:188(52) ack 233..
10.28.6.1.29447 > 192.168.2.62.22: . ack 349 win 251 (DF)..
I just want to get the first ip on each line without the port number. So from the above output I want to get just.....
10.28.6.1
10.28.6.1
10.28.6.1
10.28.6.1
Thanks
try this cut line:
cut -f1-4 -d "."
test with your data:
kent$ echo "10.28.6.1.29447 > 192.168.2.62.22: P 3758838950:3758839034..
10.28.6.1.29447 > 192.168.2.62.22: P 84:136(52) ack 117..
10.28.6.1.29447 > 192.168.2.62.22: P 136:188(52) ack 233..
10.28.6.1.29447 > 192.168.2.62.22: . ack 349 win 251 (DF).."|cut -f1-4 -d "."
10.28.6.1
10.28.6.1
10.28.6.1
10.28.6.1
perl -lane '$F[0]=~s/(.*)\..*/$1/g;print $F[0]' your_file