Write response requests on logs with Nginx - amazon-web-services

Question

!UPDATE!
Big thank's to #ivan
default.conf:
log_format include_id '$remote_addr - $remote_user [$time_local] $request_id "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent"';
server {
listen 443;
listen [::]:443;
ssl_certificate /etc/nginx/conf.d/cert.pem;
ssl_certificate_key /etc/nginx/conf.d/key.pem;
location / {
proxy_pass http://sns-mock:9911;
proxy_store /var/log/nginx/requests/mohsin.json;
}
}
docker-compose.yml:
version: '3.7'
services:
sns-mock:
image: s12v/sns
container_name: sns-mock
ports:
- "9911:9911"
networks:
- default-nw
volumes:
- ./config/db.json:/etc/sns/db.json
sns-mock-proxy:
image: nginx:latest
container_name: sns-mock-proxy
ports:
- "8443:443"
networks:
- default-nw
volumes:
- ./conf.d:/etc/nginx/conf.d
- ./log/nginx:/var/log/nginx
- ./log/nginx/requests:/var/log/nginx/requests
depends_on:
- sns-mock
networks:
default-nw:
external: true
Php test file:
<?php
use Aws\Result;
use Aws\Sns\SnsClient;
use Illuminate\Http\JsonResponse;
use PHPUnit\Framework\TestCase;
class FooTest extends TestCase
{
public function testFoo()
{
$snsClient = new SnsClient([
'endpoint' => 'localhost:8443',
'region' => 'eu-west-2',
'version' => 'latest'
]);
$result = $snsClient->publish([
'Message' => 'foo',
'TopicArn' => 'arn:aws:sns:eu-west-2:123450000001:test-topic'
]);
dd($result, 'ok');
}
}
Test result:
My project:
On my access.log file, i can catch response from phpunit test like here:
172.28.0.1 - - [23/May/2022:06:54:04 +0000] "POST / HTTP/1.1" 200 270 "-" "aws-sdk-php/3.222.17 OS/Linux/5.13.0-41-generic lang/php/8.1.6 GuzzleHttp/7" "-"
===================================================================================================================================

Answer 1

The general answer to your question is "No". However some kind of workaround is possible.
Every request gets processed by nginx receives some internal request ID available via the $request_id variable (16 random bytes, in hexadecimal). You can add that ID to your access log defining your own custom log format:
log_format include_id '$remote_addr - $remote_user [$time_local] $request_id "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent"';
access_log /var/log/nginx/access.log include_id;
Next, in the same location where you have your proxy_pass directive add the proxy_store one:
proxy_store /var/log/nginx/requests/$request_id.json;
I use the separate /var/log/nginx/requests directory here to not make your /var/log/nginx turning into a mess. Of course it should be created manually before you start the docker container, and it should be writable the same way as /var/log/nginx itself (including such a things as SELinux context, if being used on the host system). However for the testing purposes you can start with the proxy_store /var/log/nginx/$request_id.json;.
The whole nginx config should look like
log_format include_id '$remote_addr - $remote_user [$time_local] $request_id "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent"';
server {
listen 443;
listen [::]:443;
access_log /var/log/nginx/access.log include_id;
ssl_certificate /etc/nginx/conf.d/cert.pem;
ssl_certificate_key /etc/nginx/conf.d/key.pem;
location / {
proxy_pass http://sns-mock:9911;
proxy_store /var/log/nginx/requests/$request_id.json;
}
}
If you want requests to be available via the plain HTTP protocol too, you can use the same configuration for the plain HTTP server block instead of the one you've shown in your question:
server {
listen 80;
listen [::]:80;
access_log /var/log/nginx/access.log include_id;
location / {
proxy_pass http://sns-mock:9911;
proxy_store /var/log/nginx/requests/$request_id.json;
}
}
or issue an HTTP-to-HTTPS redirect instead:
server {
listen 80;
listen [::]:80;
return 308 https://$host$request_uri;
}
Now the response body for each request which can be identified from the access log via its request ID, something like
172.18.0.1 - - [20/May/2022:19:54:14 +0000] d6010d713b2dce3cd2713f1ea178e140 "POST / HTTP/1.1" 200 615 "-" "aws-sdk-php/3.222.17 OS/Linux/5.13.0-41-generic lang/php/8.1.6 GuzzleHttp/7" "-"
will be available under the /var/log/nginx/requests directory (response body for the request shown in the given access log entry will be available as d6010d713b2dce3cd2713f1ea178e140.json file).