What is the right regex on kusto query language (KQL) to extract just the last part of a messages table? - regex

I'm trying to extract the last part of a message using KQL, the patter is consistent on what part of the message is needed, for example, I need to extract everything next to ">]" characters.
In the followed example, would be "Connection Timeout Expired. The timeout period elapsed during the post-login phase.":
System.ComponentModel.SQLClientException (258): Unknown error 258 [Open]
at System.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject,
DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
[<GUI:03c09dfd-678f-45a1-a446-05f5217a705f>] Connection Timeout Expired. The timeout period elapsed during the post-login phase.
I've been trying suing the followed regex patter but it fails with the message: SemanticException.
traces
| where ...
| project extract(#"(?<=>])",1,message)
Please someone advice.

you could use the parse operator.
for example:
print input = ```System.ComponentModel.SQLClientException (258): Unknown error 258 [Open]
at System.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject,
DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
[<GUI:03c09dfd-678f-45a1-a446-05f5217a705f>] Connection Timeout Expired. The timeout period elapsed during the post-login phase.```
| parse input with * ">] " output
input
output
System.ComponentModel.SQLClientException (258): Unknown error 258 [Open] at System.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e) at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) [GUI:03c09dfd-678f-45a1-a446-05f5217a705f] Connection Timeout Expired. The timeout period elapsed during the post-login phase.
Connection Timeout Expired. The timeout period elapsed during the post-login phase.

Related

Kestrel gives bad request after FIN and end of request

I have a API made in Dotnet 6, running in a docker container and hosted on AWS.
The client is built in Wordpress and uses php curl to make requests to the api.
The Api is built with using minimal APIs and as much as I can see it doesn't do anything too fancy.
After going live with the API I did start notice strange logs where requests end with a BadHttpRequestException.
I have not been able to trace what is causing it but if we look at the server logs that traces the entire request we can see that the request is fulfilled and it sends back the response with a status of 200.
After that at row 18 it starts to close the connection, but at row 19 it starts to read the body again which after a while ends with a Bad request error at row 22 before the connection is stopped.
This error is not caught in my exception middleware so I'm lost to what to do or how to fix in.
Any suggestions would be highly appreciated.
Both Logs below are for the same endpoint on the api
Log with error in it
1 "Connection id \"{ConnectionId}\" accepted.",
2 "Connection id \"{ConnectionId}\" started.",
3 "{HostingRequestStartingLog:l}",
4 "{CandidateCount} candidate(s) found for the request path '{Path}'",
5 "Request matched endpoint '{EndpointName}'",
6 "The request has an origin header: '{origin}'.",
7 "CORS policy execution successful.",
8 "Successfully validated the token.",
9 "AuthenticationScheme: {AuthenticationScheme} was successfully authenticated.",
10"AuthenticationScheme: {AuthenticationScheme} was successfully authenticated.",
11"Authorization was successful.",
12"Executing endpoint '{EndpointName}'",
13"Writing value of type '{Type}' with status code '200'.",
14"Executed endpoint '{EndpointName}'",
15"HTTP {RequestMethod} {RequestPath} responded 200 in {Elapsed:0.0000} ms",
16"Connection id \"{ConnectionId}\" completed keep alive response.",
17"{HostingRequestFinishedLog:l}",
18"Connection id \"{ConnectionId}\" received FIN.",
19"Connection id \"{ConnectionId}\", Request id \"{TraceIdentifier}\": started reading request body.",
20"Connection id \"{ConnectionId}\" sending FIN because: \"{Reason}\"",
21"Connection id \"{ConnectionId}\" disconnecting.",
22"Connection id \"{ConnectionId}\" bad request data: \"{message}\"",
23"Exception": "Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.\n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.TryReadInternal(ReadResult& readResult)\n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.OnConsumeAsync()"
24"Connection id \"{ConnectionId}\", Request id \"{TraceIdentifier}\": done reading request body.",
25"Connection id \"{ConnectionId}\" stopped.",
Log that is fine
1 "Connection id \"{ConnectionId}\" accepted.",
2 "Connection id \"{ConnectionId}\" started.",
3 "{HostingRequestStartingLog:l}",
4 "{CandidateCount} candidate(s) found for the request path '{Path}'",
5 "Request matched endpoint '{EndpointName}'",
6 "The request has an origin header: '{origin}'.",
7 "CORS policy execution successful.",
8 "Successfully validated the token.",
9 "AuthenticationScheme: {AuthenticationScheme} was successfully authenticated.",
10"AuthenticationScheme: {AuthenticationScheme} was successfully authenticated.",
11"Authorization was successful.",
12"Executing endpoint '{EndpointName}'",
13"Writing value of type '{Type}' with status code '200'.",
14"Executed endpoint '{EndpointName}'",
15"HTTP {RequestMethod} {RequestPath} responded 200 in {Elapsed:0.0000} ms",
16"Connection id \"{ConnectionId}\" completed keep alive response.",
17"{HostingRequestFinishedLog:l}",
18"Connection id \"{ConnectionId}\", Request id \"{TraceIdentifier}\": started reading request body.",
19"Connection id \"{ConnectionId}\", Request id \"{TraceIdentifier}\": done reading request body.",
20"Connection id \"{ConnectionId}\" received FIN.",
21"Connection id \"{ConnectionId}\" disconnecting.",
22"Connection id \"{ConnectionId}\" sending FIN because: \"{Reason}\"",
23"Connection id \"{ConnectionId}\" stopped.",

fluentd regexp to extract events from a log file

I'm new to fluentd.
I have a log that I want to push to AWS with fluentd but I can't figure out what the regexp should be.
All the log lines, except the multilines, start with a UUID.
Here's a sample log:
6b0815f2-8ff1-4181-a4e6-058148288281 2020-11-03 13:00:05.976366 [DEBUG] switch_core_state_machine.c:611 (some_other_data) State Change CS_REPORTING -> CS_DESTROY
And, I'm trying to get UUID, DateTime, and Message.
With this regex:
/^(?<UUID>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) (?<time>.*) (?<message>[^ ]*)/gm
I'm getting the last word CS_DESTROY.
I tried fluentular and still got:
text:
f6a6e1ae-e52e-4aba-a8a5-4e3cc7f40914 2020-11-03 14:32:34.975779 [CRIT] mod_dptools.c:1866 audio3: https://mydomain.s3-eu-west-1.amazonaws.com/media/576d06e5-04fc-11eb-a52c-020fd8c14d18/5f9ddf2d5df0f698094395.mpg
regexp:
^(?<UUID>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) (?<time>.*) (?<message>[^ ]*)$
and got:
time 2020/11/03 14:32:34 +0000
UUID f6a6e1ae-e52e-4aba-a8a5-4e3cc7f40914
message https://mydomain.s3-eu-west-1.amazonaws.com/media/576d06e5-04fc-11eb-a52c-020fd8c14d18/5f9ddf2d5df0f698094395.mpg
It's missing what's between the datetime and "https".
Try:
^(?<UUID>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) (?<time>[^\[]*) (?<message>\[.*)$
Live at rubular: https://rubular.com/r/JQQXs5VTkr2IxM
Here's the output for both logs:
Match 1
UUID 6b0815f2-8ff1-4181-a4e6-058148288281
time 2020-11-03 13:00:05.976366
message [DEBUG] switch_core_state_machine.c:611 (some_other_data) State Change CS_REPORTING -> CS_DESTROY
Match 2
UUID f6a6e1ae-e52e-4aba-a8a5-4e3cc7f40914
time 2020-11-03 14:32:34.975779
message [CRIT] mod_dptools.c:1866 audio3: https://mydomain.s3-eu-west-1.amazonaws.com/media/576d06e5-04fc-11eb-a52c-020fd8c14d18/5f9ddf2d5df0f698094395.mpg

Regular expression for nxlog

I have a set of logs which do not have a fixed line number.
I am trying to get these logs to Elasticsearch through nxlog.
------------------------------------------------------
Timestamp: 2020-03-03 00:05:04Z|User -DEFAULT|Cart -DEFAULT|Source -mscorlib|Message -StartIndex cannot be less than zero
------------------------------------------------------
------------------------------------------------------
Timestamp: 2020-03-03 03:05:53Z|User -Simulate Pricing | Initiate | |Cart -|Level -INFO|Severity -Information|Class -SalesOrderClient.vb|Function - Simulate(testVal As Object) Method :| Cart_ID: 8092662
Timestamp: 2020-03-03 00:00:30Z|User -|Cart -|Source -mscorlib|Message -Thread was being aborted.|StackTrace - at System.Threading.Thread.AbortInternal()
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
------------------------------------------------------
------------------------------------------------------
Timestamp: 2020-03-03 00:18:57Z|User -DEFAULT|Cart -DEFAULT|Source -mscorlib|Message -StartIndex cannot be less than zero.
------------------------------------------------------
This has a few logs with multiline (no fixedlines) and a few with a single line.
When im trying to fetch this data with nxlog, my third log is getting split into 4 different rows for each new line present.
How can I avoid this issue. What Regular expression can I use here? I want one full log to be in a single row, irrespective of how many line there are. The lines are present in the logs.
Thanks in Advance.
Use a regexp for HeaderLine with the xm_multiline module that matches your timestamp.
See the examples at https://nxlog.co/documentation/nxlog-user-guide/xm_multiline.html

Package Manager getting error Failed to generate a user instance of SQL Server due to a failure in starting the process for the user instance

While i use update-database command in nuget package manager.
getting
"Failed to generate a user instance of SQL Server due to a failure in starting the process for the user instance. The connection will be closed."
i check SQL Server configuration
2 check SSMS login
Working properly
For more details please check complete log.
PM> update -database
update : The term 'update' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ update -database
+ ~~~~~~
+ CategoryInfo : ObjectNotFound: (update:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
PM> update-database
Specify the '-Verbose' flag to view the SQL statements being applied to the target database.
System.Data.SqlClient.SqlException (0x80131904): Failed to generate a user instance of SQL Server due to a failure in starting the process for the user instance. The connection will be closed.
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at System.Data.Entity.Infrastructure.Interception.DbConnectionDispatcher.b__36(DbConnection t, DbConnectionInterceptionContext c)
at System.Data.Entity.Infrastructure.Interception.InternalDispatcher1.Dispatch[TTarget,TInterceptionContext](TTarget target, Action2 operation, TInterceptionContext interceptionContext, Action3 executing, Action3 executed)
at System.Data.Entity.Infrastructure.Interception.DbConnectionDispatcher.Open(DbConnection connection, DbInterceptionContext interceptionContext)
at System.Data.Entity.SqlServer.SqlProviderServices.<>c__DisplayClass33.b__32()
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.<>c__DisplayClass1.b__0()
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func1 operation)
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute(Action operation)
at System.Data.Entity.SqlServer.SqlProviderServices.UsingConnection(DbConnection sqlConnection, Action1 act)
at System.Data.Entity.SqlServer.SqlProviderServices.UsingMasterConnection(DbConnection sqlConnection, Action1 act)
at System.Data.Entity.SqlServer.SqlProviderServices.CreateDatabaseFromScript(Nullable1 commandTimeout, DbConnection sqlConnection, String createDatabaseScript)
at System.Data.Entity.SqlServer.SqlProviderServices.DbCreateDatabase(DbConnection connection, Nullable1 commandTimeout, StoreItemCollection storeItemCollection)
at System.Data.Entity.Core.Common.DbProviderServices.CreateDatabase(DbConnection connection, Nullable1 commandTimeout, StoreItemCollection storeItemCollection)
at System.Data.Entity.Core.Objects.ObjectContext.CreateDatabase()
at System.Data.Entity.Migrations.Utilities.DatabaseCreator.Create(DbConnection connection)
at System.Data.Entity.Migrations.DbMigrator.EnsureDatabaseExists(Action mustSucceedToKeepDatabase)
at System.Data.Entity.Migrations.Infrastructure.MigratorBase.EnsureDatabaseExists(Action mustSucceedToKeepDatabase)
at System.Data.Entity.Migrations.DbMigrator.Update(String targetMigration)
at System.Data.Entity.Migrations.Infrastructure.MigratorBase.Update(String targetMigration)
at System.Data.Entity.Migrations.Design.ToolingFacade.UpdateRunner.RunCore()
at System.Data.Entity.Migrations.Design.ToolingFacade.BaseRunner.Run()
ClientConnectionId:1fc5ad95-2b73-4e43-bce3-ebee4805c3fe
Error Number:15372,State:1,Class:14
Failed to generate a user instance of SQL Server due to a failure in starting the process for the user instance. The connection will be closed.
PM>
it's work for me
Deleted the contents of the following directory: c:\Users\\AppData\Local\Microsoft\Microsoft SQL Server Data\SQLEXPRESS.
and run command
PM> update-database

Sending Emails with attachment of less than 10 MB from AWS SES

We are using AWS SES to send Emails from our C# application. We have an attachment(PDF File) of size 9.28MB and when we try to send the Email with the attachment, it throws the following exception:
System.Xml.XmlException: Root element is missing. at System.Xml.XmlTextReaderImpl.Throw(Exception e) at System.Xml.XmlTextReaderImpl.ParseDocumentContent() at Amazon.Runtime.Internal.Transform.UnmarshallerContext.Read() at Amazon.Runtime.Internal.Transform.ErrorResponseUnmarshaller.Unmarshall(UnmarshallerContext context) at Amazon.SimpleEmail.Model.Transform.SendRawEmailResponseUnmarshaller.UnmarshallException(UnmarshallerContext context, Exception innerException, HttpStatusCode statusCode) at Amazon.Runtime.AmazonWebServiceClient.processWebException[X,Y](String requestName, WebException we, HttpWebRequest webRequest, IResponseUnmarshaller`2 unmarshaller, IRequest`1 request, Int32 retries) at Amazon.Runtime.AmazonWebServiceClient.Invoke[X,Y](IRequest`1 request, AbstractAWSSigner signer, IResponseUnmarshaller`2 unmarshaller) at Amazon.SimpleEmail.AmazonSimpleEmailServiceClient.SendRawEmail(SendRawEmailRequest sendRawEmailRequest) at MyApp.AmazonSESWrapper.SendRawEmail(String awsAccessKey, String awsSecretKey, List`1 to, List`1 cc, List`1 bcc, String senderEmailAddress, String replyToEmailAddress, String subject, String body, String text, String filePath, String exceptionDetails)
But when we send a 7.50MB word document as attachment, we are getting the following exception:
Amazon.SimpleEmail.AmazonSimpleEmailServiceException: Message length is more than 10485760 bytes long: '10788624'. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request. at System.Net.HttpWebRequest.GetResponse() at Amazon.Runtime.AmazonWebServiceClient.Invoke[X,Y](IRequest`1 request, AbstractAWSSigner signer, IResponseUnmarshaller`2 unmarshaller) --- End of inner exception stack trace --- at Amazon.Runtime.AmazonWebServiceClient.processWebException[X,Y](String requestName, WebException we, HttpWebRequest webRequest, IResponseUnmarshaller`2 unmarshaller, IRequest`1 request, Int32 retries) at Amazon.Runtime.AmazonWebServiceClient.Invoke[X,Y](IRequest`1 request, AbstractAWSSigner signer, IResponseUnmarshaller`2 unmarshaller) at Amazon.SimpleEmail.AmazonSimpleEmailServiceClient.SendRawEmail(SendRawEmailRequest sendRawEmailRequest) at
From the Discussion Forum and FAQ, it says the maximum attachment size is 10MB, but the error occurs for less than 10MB attachments also.
https://forums.aws.amazon.com/thread.jspa?messageID=346305
http://aws.amazon.com/ses/faqs/#49
Please suggest on this.
There is also overall message size limit, which is also 10M. As per http://aws.amazon.com/ses/faqs/:
Amazon SES will accept email messages up to 10 MB in size. This
includes any attachments that are part of the message.
So you have to ensure your request fits in these limits.
(I personally think its a bug that Amazon API throws "XmlException: Root element is missing." in case when the message size considerably exceeded the limit. It's totally misleading.)