The Istio documentation provides a sample istio-proxy log, as below:
[2019-03-06T09:31:27.354Z] "GET /status/418 HTTP/1.1" 418 - "-" 0 135 11 10 "-" "curl/7.60.0" "d209e46f-9ed5-9b61-bbdd-43e22662702a" "httpbin:8000" "172.30.146.73:80" outbound|8000||httpbin.default.svc.cluster.local - 172.21.13.94:8000 172.30.146.82:60290 -
The linked Envoy documentation mentions that for log format is as follow:
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"
%RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION%
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%"
"%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"\n
example from Envoy:
[2016-04-15T20:17:00.310Z] "POST /api/v1/locations HTTP/2" 204 - 154 0 226 100 "10.0.35.28"
"nsq2http" "cc21d9b0-cf5c-432b-8c7e-98aeb7988cd2" "locations" "tcp://10.0.2.1:80"
This format is different than the one used by istio-proxy.
The documentation seems out of sync with the Envoy documentation as the provided sample log cannot be mapped to the Envoy default logged format. What is the log format here? What is being logged?
Related
I am using gcloud CLI to configure my region and zone:
gcloud config set compute/region us-central1
gcloud config set compute/zone us-central1-c
But each command lasts for about 15 seconds, and I get a warning:
WARNING: Property validation for compute/region was skipped
Everything works fine, but why do I have 15 seconds delay, and a warning?
With verbose argument, the output is:
DEBUG: Running [gcloud.config.set] with arguments: [--verbosity: "debug", SECTION/PROPERTY: "compute/region", VALUE: "us-central1"]
Updated property [compute/region].
DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/GOOGLE_AACOUNT_REPLACED#cloudbuild.gserviceaccount.com/?recursive=true
DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/GOOGLE_AACOUNT_REPLACED#cloudbuild.gserviceaccount.com/?recursive=true HTTP/1.1" 200 185
DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/GOOGLE_AACOUNT_REPLACED#cloudbuild.gserviceaccount.com/token
DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/GOOGLE_AACOUNT_REPLACED#cloudbuild.gserviceaccount.com/token HTTP/1.1" 200 1050
DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/GOOGLE_AACOUNT_REPLACED#cloudbuild.gserviceaccount.com/?recursive=true
DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/GOOGLE_AACOUNT_REPLACED#cloudbuild.gserviceaccount.com/?recursive=true HTTP/1.1" 200 185
DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/GOOGLE_AACOUNT_REPLACED#cloudbuild.gserviceaccount.com/token
DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/GOOGLE_AACOUNT_REPLACED#cloudbuild.gserviceaccount.com/token HTTP/1.1" 200 1050
DEBUG: Starting new HTTPS connection (1): compute.googleapis.com:443
DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
WARNING: Property validation for compute/region was skipped.
To make the gcloud tool easier to use, the Google Cloud will try and validate the values provided, including “compute/region”. In this case, it has to fetch a full list of available regions from the API. If this fails, for whatever reason, then it will show this warning message.
One of the many reasons may be that the Compute Engine API is not enabled. It could also be a lack of authentication, although Cloud Build will have authentication enabled by default and you don't need any special permissions to run this command.
To find out what exactly is going wrong, you can try adding the --log-http parameter to your gcloud command line. This will display the full details of any interactions with the API, including any error message in the response.
In any case, this is simply a warning, and the config entry is still being updated. This happens even if the validation fails, e.g. the region does not exist. As I mentioned above, this is just a feature to help letting the user know if they make certain types of simple mistakes.
I'm looking to parse AWS Load Balancer log files stored in S3, to calculate metrics by the site URL www.example.com instead of the virtual host app/something.com. Is this possible? I'm using GoAccess.
https 2019-11-24T23:55:01.603141Z app/something.com 34.222.222.22:47121 190.61.18.156:80 0.008 0.252 0.000 200 200 191 725 "GET https://www.example.com:443/something.php HTTP/1.1" "Wget/1.18 (linux-gnu)" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-west-1:6474865788:targetgroup/mytargetgroup/be12345678 "Root=1-5ddb4567-149b7e874546754ed496" "www.example.com" "arn:aws:acm:eu-west-1:6474865788:certificate/pwdsw3455-4028-5cb7-854c-gdtr555" 0 2019-11-24T23:55:01.342000Z "waf,forward" "-" "-" "190.61.18.156:80" "200"
This will work for the line you posted, though you may want to use different delimiter if any of your fields can contain additional spaces.
awk -F'[ ]' '$3=$22$3' access.log | goaccess - -a
When I am learning Django,
The console usually hint me:
"GET /new_entry/5 HTTP/1.1" 200 466
What's the purpose of 200 466?
The 200 is the HTTP status code, which means the request completed successfully. The second number is the size of the request in bytes.
Hi i'm in need of some serious help,
I have logs that i wish to Parse using GROK but the problem i'm having is that they are not always consistent in content or spacing here are some obfuscated examples.
title_access_log:ipaddress1, ipaddress2, ipaddress3 - - [14/Nov/2017:08:30:00 +0000] "GET /url HTTP/1.1" 200 198454 - 153261 - 0000fD5b5OSuS2C7ZdhgwqYufJk:GH809 url
title_access_log:ipaddress1, ipaddress2 - - [14/Nov/2017:08:30:00 +0000] "GET /url HTTP/1.1" 200 2326 - 20482 V22843489635e0e42e864037eccb8ad4857500ea 0000BDzHfUFhjJmcs9R4-CyglGS:GH806 url
title_access_log:ipaddress1, ipaddress2 - - [14/Nov/2017:08:30:00 +0000] "POST /url HTTP/1.1" 200 30031 - 17942 - 0000PjpQluI9BZ0w4EDB9o2fow-:GH809 url
I have managed to make a GROK patterns that pull out up to time and date for logs that contain 2 IPs but i get suck going further or when trying to do logs with 3 ips.
Has anyone got any advice on how to tackle this.
i'm using Graylog is what i'm using to extract data to so i do have the option of using other formats than GROK.
If I run a Django development server and view my application in a browser, I get messages like this:
[08/Jan/2011 18:12:45] "GET / HTTP/1.1" 200 2714
[08/Jan/2011 18:12:45] "GET /static/style.css" 200 2714
[08/Jan/2011 18:12:45] "GET /content.html" 200 269
...
What does the last number on each line mean?
(I know that the number one before last is standard HTTP status message, but don't know what the following number represents)
It's the number of bytes sent, see django/core/servers/basehttp.py:
self.request_handler.log_request(self.status.split(' ',1)[0], self.bytes_sent)