I am trying to set a HTCondor batch system, but when I do condor_status it only shows the master in both the master and worker nodes. They both show this:
Name OpSys Arch State Activity LoadAv Mem
[master ip] LINUX X86_64 Unclaimed Idle 0.000 973
Total Owner Claimed Unclaimed Matched Preempting Backfill Drain
X86_64/LINUX 1 0 0 1 0 0 0 0
Total 1 0 0 1 0 0 0 0
Condor_restart on the master node works fine, but on the worker nodes yields this error:
ERROR
SECMAN:2010:Received "DENIED" from server for user unauthenticated#unmapped using no authentication method, which may imply host-based security. Our address was '[ip address of master]', and server's address was '[ip address of worker]'. Check your ALLOW settings and IP protocols.
Here are the config files:
of the master node:
CONDOR_HOST = [private ip of master]
DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD, STARTD
# to avoid user authentication
HOSTALLOW_READ = *
HOSTALLOW_WRITE = *
HOSTALLOW_ADMINISTRATOR = *
of the worker node:
CONDOR_HOST = [private ip of master]
DAEMON_LIST = MASTER, STARTD
# to avoid user authentication
HOSTALLOW_READ = *
HOSTALLOW_WRITE = *
HOSTALLOW_ADMINISTRATOR = *
I am allowing on the same security group:
All TCP TCP 0 - 65535
All ICMP-IPv4 All
SSH on port 22
This is how it looks like (security group ending in '6')
Apparently the issue was running condor_reconfig -full. I just reinstalled it without doing that and using systemctl restart condor instead and it worked. If someone wants to bring some insight on why it was so please do so :)
I'm using this uwsgi.ini to run a Django application.
[uwsgi]
http-socket = :8000
enable-proxy-protocol = true
chdir = /usr/local/src/api
module = api.wsgi
uid = root
gid = root
pidfile = /var/run/api-uwsgi.pid
master = true
processes = 10
chmod-socket = 664
threaded-logger = true
logto = /var/log/api/uwsgi.log
log-maxsize = 10000000
logfile-chown = true
vacuum = true
die-on-term = true
I've added an API url to perform database and cache health checks under the /health-check url. This API returns status code 200 if everything is fine. Now I want to be able to health check in layer 7 using this API with HAProxy but using option httpchk the response status code is 301, so the health check fails. Here is the backend part of my HAProxy config.
backend http_server
mode http
balance leastconn
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk
http-check send meth GET uri /health-check ver HTTP/1.1 hdr Accept application\json
http-check expect rstatus 200
server app1 192.168.0.11:8000 check inter 500 downinter 5s fall 2 rise 3
server app2 192.168.0.12:8000 check inter 500 downinter 5s fall 2 rise 3
Here is the result of running the Django apps with uWSGI and HAProxy. Note that the health check on layer 4 is working as expected.
Server http_server/app2 is DOWN, reason: Layer7 wrong status, code: 301, info: "Moved Permanently", check duration: 54ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
So what is causing this problem and is there a way to fix it?
There was two steps to solve this problem:
If no url route matches with the requested one Django's APPEND_SLASH setting appends a / to to end of the url and performs a 301 redirect on that. I've used /health-check instead of /health-check/ so the request was redirected from /health-check to /health-check/
The hdr option of http-check accepts a key and a value. For the accept header the value must be inside quotes so hdr Accept application\json becomes hdr accept "application/json"
Here is the final and working part of the backend config:
backend http_server
mode http
balance leastconn
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk
http-check send meth GET uri /health-check/ ver HTTP/1.1 hdr accept "application/json"
http-check expect rstatus 200
server 192.168.0.11:8000 check inter 500 downinter 2s fall 2 rise 3
server 192.168.0.12:8000 check inter 500 downinter 2s fall 2 rise 3
We are trying to get X-Ray trace data from a local dotnet core 3.1 app sending trace data to a local X-Ray Daemon. As a start, we've created a generic web api and added swagger (just to make testing easier).
Startup.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.HttpsPolicy;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.OpenApi.Models;
using Amazon.XRay.Recorder.Core;
using log4net;
using log4net.Config;
using System.Reflection;
using System.IO;
using Amazon;
using System.Net;
using Amazon.XRay.Recorder.Core.Internal.Utils;
using Amazon.XRay.Recorder.Core.Sampling.Local;
namespace AWS_XRay
{
public class Startup
{
public static ILog log;
static Startup() // create log4j instance
{
var logRepository = LogManager.GetRepository(Assembly.GetEntryAssembly());
XmlConfigurator.Configure(logRepository, new FileInfo("log4net.config"));
log = LogManager.GetLogger(typeof(Startup));
AWSXRayRecorder.RegisterLogger(LoggingOptions.Log4Net);
}
public Startup(IConfiguration configuration)
{
Configuration = configuration;
Environment.SetEnvironmentVariable("AWS_XRAY_DAEMON_ADDRESS", "127.0.0.1:2000");
Environment.SetEnvironmentVariable("AWS_XRAY_CONTEXT_MISSING", "LOG_ERROR");
var recorder = new AWSXRayRecorderBuilder().WithSamplingStrategy(newLocalizedSamplingStrategy("sampling-rules.json")).Build();
AWSXRayRecorder.InitializeInstance(configuration, recorder);
}
public IConfiguration Configuration { get; }
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.AddControllers();
// Register the Swagger generator, defining 1 or more Swagger documents
services.AddSwaggerGen(c =>
{
c.SwaggerDoc("v1", new OpenApiInfo { Title = "My API", Version = "v1" });
});
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
app.UseXRay("WeatherForecast");
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
app.UseSwagger();
app.UseSwaggerUI(c =>
{
c.SwaggerEndpoint("/swagger/v1/swagger.json", "My API V1");
c.RoutePrefix = string.Empty;
});
app.UseHttpsRedirection();
app.UseRouting();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
});
}
}
}
Then we decorated the controller with the relevant or what we think is relevant
WeatherController
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Amazon.XRay.Recorder.Core;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;
namespace AWS_XRay.Controllers
{
[ApiController]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
private static readonly string[] Summaries = new[]
{
"Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
};
private readonly ILogger<WeatherForecastController> _logger;
public WeatherForecastController(ILogger<WeatherForecastController> logger)
{
_logger = logger;
}
[HttpGet]
[Route("GetWeather")]
public async Task<IActionResult> WeatherForecast()
{
AWSXRayRecorder.Instance.BeginSegment("weatherget"); // generates `TraceId` for you
try
{
var rng = new Random();
var result = Enumerable.Range(1, 5).Select(index => new WeatherForecast
{
Date = DateTime.Now.AddDays(index),
TemperatureC = rng.Next(-20, 55),
Summary = Summaries[rng.Next(Summaries.Length)]
})
.ToArray();
// can create custom subsegments
return Ok(result);
}
catch (Exception e)
{
AWSXRayRecorder.Instance.AddException(e);
return StatusCode(500, e);
}
finally
{
AWSXRayRecorder.Instance.EndSegment();
}
}
}
}
When running the application, looking at the logs. This is what we see...
*sdk-log.txt"
2020-04-14 16:04:21,740 [1] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Initializing with custom sampling configuration : sampling-rules.json
2020-04-14 16:04:22,035 [1] DEBUG Amazon.XRay.Recorder.Core.Internal.Utils.IPEndPointExtension - Determined that 127.0.0.1:2000 is an IP.
2020-04-14 16:04:22,039 [1] INFO Amazon.XRay.Recorder.Core.Internal.Utils.IPEndPointExtension - Using custom daemon address for UDP and TCP: 127.0.0.1:2000
2020-04-14 16:04:22,042 [1] DEBUG Amazon.XRay.Recorder.Core.Strategies.DefaultExceptionSerializationStrategy - Setting max stack frame size : 50
2020-04-14 16:04:22,073 [1] DEBUG Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl - Context missing mode : RUNTIME_ERROR
2020-04-14 16:04:22,073 [1] DEBUG Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl - AWS_XRAY_CONTEXT_MISSING environment variable is set to LOG_ERROR. Override local value.
2020-04-14 16:04:22,078 [1] DEBUG Amazon.XRay.Recorder.Core.Internal.Utils.IPEndPointExtension - Determined that 127.0.0.1:2000 is an IP.
2020-04-14 16:04:22,078 [1] INFO Amazon.XRay.Recorder.Core.Internal.Utils.IPEndPointExtension - Using custom daemon address for UDP and TCP: 127.0.0.1:2000
2020-04-14 16:04:22,078 [1] DEBUG Amazon.XRay.Recorder.Core.Strategies.DefaultExceptionSerializationStrategy - Setting max stack frame size : 50
2020-04-14 16:04:22,078 [1] DEBUG Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl - Context missing mode : RUNTIME_ERROR
2020-04-14 16:04:22,078 [1] DEBUG Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl - AWS_XRAY_CONTEXT_MISSING environment variable is set to LOG_ERROR. Override local value.
2020-04-14 16:04:22,078 [1] DEBUG Amazon.XRay.Recorder.Core.AWSXRayRecorder - Using custom X-Ray recorder.
2020-04-14 16:04:22,079 [1] DEBUG Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl - Context missing mode : RUNTIME_ERROR
2020-04-14 16:04:22,080 [1] DEBUG Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl - AWS_XRAY_CONTEXT_MISSING environment variable is set to LOG_ERROR. Override local value.
2020-04-14 16:04:22,899 [4] DEBUG Amazon.XRay.Recorder.Handlers.AspNetCore.Internal.AWSXRayMiddleware - Trace header doesn't exist or not valid : (). Injecting a new one.
2020-04-14 16:04:22,911 [4] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = localhost, path = /index.html, method = GET
2020-04-14 16:04:23,393 [4] DEBUG Amazon.XRay.Recorder.Handlers.AspNetCore.Internal.AWSXRayMiddleware - Trace header doesn't exist or not valid : (). Injecting a new one.
2020-04-14 16:04:23,394 [4] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = localhost, path = /swagger/v1/swagger.json, method = GET
2020-04-14 16:04:27,497 [4] DEBUG Amazon.XRay.Recorder.Handlers.AspNetCore.Internal.AWSXRayMiddleware - Trace header doesn't exist or not valid : (). Injecting a new one.
2020-04-14 16:04:27,499 [4] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = localhost, path = /WeatherForecast/GetWeather, method = GET
2020-04-14 16:04:27,602 [4] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = , path = , method =
2020-04-14 16:04:29,740 [4] DEBUG Amazon.XRay.Recorder.Handlers.AspNetCore.Internal.AWSXRayMiddleware - Trace header doesn't exist or not valid : (). Injecting a new one.
2020-04-14 16:04:29,741 [4] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = localhost, path = /WeatherForecast/GetWeather, method = GET
2020-04-14 16:04:29,745 [4] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = , path = , method =
2020-04-14 16:04:30,149 [13] DEBUG Amazon.XRay.Recorder.Handlers.AspNetCore.Internal.AWSXRayMiddleware - Trace header doesn't exist or not valid : (). Injecting a new one.
2020-04-14 16:04:30,150 [13] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = localhost, path = /WeatherForecast/GetWeather, method = GET
2020-04-14 16:04:30,152 [13] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = , path = , method =
2020-04-14 16:04:30,346 [4] DEBUG Amazon.XRay.Recorder.Handlers.AspNetCore.Internal.AWSXRayMiddleware - Trace header doesn't exist or not valid : (). Injecting a new one.
2020-04-14 16:04:30,346 [4] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = localhost, path = /WeatherForecast/GetWeather, method = GET
2020-04-14 16:04:30,349 [4] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = , path = , method =
2020-04-14 16:04:30,517 [13] DEBUG Amazon.XRay.Recorder.Handlers.AspNetCore.Internal.AWSXRayMiddleware - Trace header doesn't exist or not valid : (). Injecting a new one.
2020-04-14 16:04:30,518 [13] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = localhost, path = /WeatherForecast/GetWeather, method = GET
2020-04-14 16:04:30,529 [13] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Found a matching rule : (hostToMatch=*, httpMethodToMatch=Get, urlPathToMatch=*, fixedTarget=0, rate=0, description=Weather) for host = , path = , method =
2020-04-14 16:30:02,682 [1] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Initializing with custom sampling configuration : sampling-rules.json
Question 1
Based on the output in the config file, is there any trace data being sent to the daemon? We can't see any errors from the output, log level is set to DEBUG. Can't definitively say it is sending trace data to although no errors.
Daemon Config & Logs
cfg.yaml
# Maximum buffer size in MB (minimum 3). Choose 0 to use 1% of host memory.
TotalBufferSizeMB: 0
# Maximum number of concurrent calls to AWS X-Ray to upload segment documents.
Concurrency: 8
# Send segments to AWS X-Ray service in a specific region
Region: "eu-west-1"
# Change the X-Ray service endpoint to which the daemon sends segment documents.
Endpoint: "xray.eu-west-1.amazonaws.com"
Socket:
# Change the address and port on which the daemon listens for UDP packets containing segment documents.
UDPAddress: "127.0.0.1:2000"
# Change the address and port on which the daemon listens for HTTP requests to proxy to AWS X-Ray.
TCPAddress: "127.0.0.1:2000"
Logging:
LogRotation: true
# Change the log level, from most verbose to least: dev, debug, info, warn, error, prod (default).
LogLevel: "dev"
# Output logs to the specified file path.
LogPath: "xray.log"
# Turn on local mode to skip EC2 instance metadata check.
LocalMode: true
# Amazon Resource Name (ARN) of the AWS resource running the daemon.
ResourceARN: ""
# Assume an IAM role to upload segments to a different account.
RoleARN: "************************"
# Disable TLS certificate verification.
NoVerifySSL: false
# Upload segments to AWS X-Ray through a proxy.
ProxyAddress: ""
# Daemon configuration file format version.
Version: 2
Looking at the log file
2020-04-14T16:35:40+02:00 [Debug] Segment batch: done!
2020-04-14T16:35:40+02:00 [Debug] Skipped telemetry data as no segments found
2020-04-14T16:35:40+02:00 [Debug] telemetry: done!
2020-04-14T16:35:40+02:00 [Debug] Segment batch: done!
2020-04-14T16:35:40+02:00 [Debug] Segment batch: done!
2020-04-14T16:35:40+02:00 [Debug] Segment batch: done!
2020-04-14T16:35:40+02:00 [Debug] Segment batch: done!
2020-04-14T16:35:40+02:00 [Debug] Segment batch: done!
2020-04-14T16:35:40+02:00 [Debug] Segment batch: done!
2020-04-14T16:35:40+02:00 [Debug] Segment batch: done!
2020-04-14T16:35:40+02:00 [Debug] processor: done!
2020-04-14T16:35:40+02:00 [Debug] Trace segment: received: 0, truncated: 0, processed: 0
2020-04-14T16:35:40+02:00 [Debug] Shutdown finished. Current epoch in nanoseconds: 1586874940496183800
2020-04-14T16:35:42+02:00 [Info] Initializing AWS X-Ray daemon 3.2.0
2020-04-14T16:35:42+02:00 [Debug] Listening on UDP 127.0.0.1:2000
2020-04-14T16:35:42+02:00 [Info] Using buffer memory limit of 80 MB
2020-04-14T16:35:42+02:00 [Info] 1280 segment buffers allocated
2020-04-14T16:35:42+02:00 [Debug] Using Endpoint read from Config file: xray.eu-west-1.amazonaws.com
2020-04-14T16:35:42+02:00 [Debug] Using proxy address:
2020-04-14T16:35:42+02:00 [Debug] Fetch region eu-west-1 from commandline/config file
2020-04-14T16:35:42+02:00 [Info] Using region: eu-west-1
2020-04-14T16:35:42+02:00 [Debug] ARN of the AWS resource running the daemon:
2020-04-14T16:35:42+02:00 [Debug] No Metadata set for telemetry records
2020-04-14T16:35:42+02:00 [Debug] Using Endpoint: https://xray.eu-west-1.amazonaws.com
2020-04-14T16:35:42+02:00 [Debug] Telemetry initiated
2020-04-14T16:35:42+02:00 [Info] HTTP Proxy server using X-Ray Endpoint : xray.eu-west-1.amazonaws.com
2020-04-14T16:35:42+02:00 [Debug] Using Endpoint: https://xray.eu-west-1.amazonaws.com
2020-04-14T16:35:42+02:00 [Debug] Batch size: 50
Question 2
Looking at the log file of the daemon, the line Trace segment: received: 0, truncated: 0, processed: 0 seems to indicate that it never received trace data? Why not, what are we missing? I'm suspecting that we are not instrumenting the application properly, but not sure.
For anyone that's interested. Herewith the solution to the problem (actually multiple problems)
Step 1 - Startup File Code
public Startup(IConfiguration configuration)
{
AWSXRayRecorder.InitializeInstance(configuration: Configuration); // Inititalizing Configuration object with X-Ray recorder
AWSSDKHandler.RegisterXRayForAllServices(); // All AWS SDK requests will be traced
}
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
//Make sure this is after env.IsDevelopment()
app.UseXRay("WeatherForecast");
.....
}
Make sure appsettings.json and sampling-rules.json mimic's the Sample App
Once the code runs, the log file of the app would look something like this.
I felt that the AWS.SDK package generates a lot of noise even when using the Sample App, which I omitted here. That said, DEBUG logs tend to be that way.
2020-04-15 11:34:04,262 [5] INFO Amazon.XRay.Recorder.Core.Internal.Utils.DaemonConfig - The given daemonAddress () is invalid, using default daemon UDP and TCP address 127.0.0.1:2000.
2020-04-15 11:34:04,368 [5] INFO Amazon.Runtime.Internal.RuntimePipelineCustomizerRegistry - Applying runtime pipeline customization X-Ray Registration Customization
2020-04-15 11:34:04,389 [5] INFO Amazon.XRay.Recorder.Core.Sampling.DefaultSamplingStrategy - No effective centralized sampling rule match. Fallback to local rules.
2020-04-15 11:34:04,390 [5] DEBUG Amazon.XRay.Recorder.Core.Sampling.Local.LocalizedSamplingStrategy - Can't match a rule for host = localhost, path = /index.html, method = GET
2020-04-15 11:34:04,573 [5] DEBUG **Amazon.XRay.Recorder.Core.Internal.Emitters.UdpSegmentEmitter - UDP Segment emitter endpoint: 127.0.0.1:2000.**
Ultimately, you are looking for the last line Amazon.XRay.Recorder.Core.Internal.Emitters.UdpSegmentEmitter - UDP Segment emitter endpoint: 127.0.0.1:2000.
Step 2 - Configure the Daemon
If you install the Daemon as a Windows Service locally. I ran into a couple of additional problems.
A - It doesn't put everything in one place and it doesn't look at the configuration file that it extracted. Unless you put the cfg.yaml file in System32.
B - The service probably won't have access to the .aws folder where the credentials are stored.
I fixed problems A, by doing the following (i'm sure you could achieve the same goal in multiple ways)
Since i'm not a powershell expert, I just moved the extracted content to a folder of my choosing and modified the service path in the registry to point to that folder as well as added the appropriate flags so that it logs to the location you expect as well as use the cfg.yaml file you expect.
regedit -> Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AWSXRayDaemon
Set image path with flags -f for log file and -c for config file
C:\YOUR USER\.aws\aws-xray-daemon\xray.exe -f C:\YOUR USER\.aws\aws-xray-daemon\xray-daemon.log -c C:\YOUR USER\.aws\aws-xray-daemon\cfg.yaml
The last problem was the Daemon not having the appropriate permissions to access the credentials file inside the .aws folder.
Log file will look something like this
2020-04-15T09:35:54+02:00 [Debug] processor: sending partial batch
2020-04-15T09:35:54+02:00 [Debug] processor: segment batch size: 1. capacity: 50
2020-04-15T09:35:54+02:00 [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2020-04-15T09:35:54+02:00 [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
The NoCredentialProviders line indicates a permission issue.
I then modified the service to run as an administrator, which solved problem B.
daemon.log
2020-04-15T09:41:31+02:00 [Debug] Received request on HTTP Proxy server : /GetSamplingRules
2020-04-15T09:41:32+02:00 [Debug] processor: sending partial batch
2020-04-15T09:41:32+02:00 [Debug] processor: segment batch size: 1. capacity: 50
2020-04-15T09:41:33+02:00 [Debug] Received request on HTTP Proxy server : /GetSamplingRules
2020-04-15T09:41:33+02:00 [Info] Successfully sent batch of 1 segments (0.871 seconds)
2020-04-15T09:41:34+02:00 [Debug] processor: sending partial batch
2020-04-15T09:41:34+02:00 [Debug] processor: segment batch size: 1. capacity: 50
2020-04-15T09:41:34+02:00 [Info] Successfully sent batch of 1 segments (0.197 seconds)
You are looking for the line successfully sent batch as confirmation that the Daemon sent the trace to the X-Ray service.
Hope this helps someone.
Cheers
By looking at the daemon logs looks like trace data is not sent to the service. I think instrumentation could be the issue. I would recommend you to read this documentation for instrumentation (https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-dotnet.html). You might have to instrument outgoing HTTP calls, incoming http request and outgoing AWS SDK calls in order to see trace view of your application. Hope this helps!
I am completely new in usung of Elastcalert. I am trying to use Elasticalert for striking email when no log is sent to logstash from my client server. I have successfully installed Elastcalert on my master server. However, when I run elastalert-create-index I get following error:
Traceback (most recent call last):
File "/usr/bin/elastalert-create-index", line 11, in <module>
load_entry_point('elastalert==0.1.21', 'console_scripts', 'elastalert-
create-index')()
File "/usr/lib/python2.7/site-packages/elastalert-0.1.21-
py2.7.egg/elastalert/create_index.py", line 77, in main
username = args.username if args.username else data.get('es_username')
UnboundLocalError: local variable 'data' referenced before assignment
My config.yaml is as follow:
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: example_rules
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
minutes: 1
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: localhost
# The Elasticsearch port
es_port: 9200
# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1
# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-
started.html
# for details
#profile: test
# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to Elasticsearch
#use_ssl: True
# Verify TLS certificates
#verify_certs: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?
highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET
# Option basic-auth username and password for Elasticsearch
#es_username:
#es_password:
# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 2
Did you try running elastalert-create-index without any arguments? It guides you through the setup process like this:
$>elastalert-create-index
Enter Elasticsearch host: localhost
Enter Elasticsearch port: 9200
Use SSL? t/f: f
Enter optional basic-auth username (or leave blank):
Enter optional basic-auth password (or leave blank):
Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request):
New index name? (Default elastalert_status)
Name of existing index to copy? (Default None)
Elastic Version:6
Mapping used for string:{'type': 'keyword'}
New index elastalert_status created
Done!