Success Email
Email SUBJECT : [ENV] Details of the message(process name)
Ex : [DEV] : WSO2 ESB INFO : XXXX_UpdatePromotionHeaderProxy Completed
DateTime :
Transaction ID :
Process Name :
File Name :
Error Email
Email SUBJECT : [ENV] Details of the message(process name)
Ex : [DEV] : WSO2 ESB ERROR : UpdatePromotionHeaderProxy FAILED
DateTime :
Transaction ID :
Process Name :
File Name :
Error Code :
Error Description :
Error Stack Trace :
Related
I launched a dataflow batch job to load csv data from GCS to Pubsub.
Dataflow job is failing with the following log portion:
Error message from worker: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "One or more messages in the publish request is empty. Each message must contain either non-empty data, or at least one attribute.", "reason" : "badRequest" } ], "message" : "One or more messages in the publish request is empty. Each message must contain either non-empty data, or at least one attribute.", "status" : "INVALID_ARGUMENT" } com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150) com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113) com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:443) com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) org.apache.beam.sdk.io.gcp.pubsub.PubsubJsonClient.publish(PubsubJsonClient.java:138) org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Write$PubsubBoundedWriter.publish(PubsubIO.java:1195) org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Write$PubsubBoundedWriter.finishBundle(PubsubIO.java:1184)
So basically it's saying that there is at least on line empty but the csv data contains some empty fields but not a full empty line
here is a sample below
2019-12-01 00:00:00 UTC,remove_from_cart,5712790,1487580005268456287,,f.o.x,6.27,576802932,51d85cb0-897f-48d2-918b-ad63965c12dc
2019-12-01 00:00:00 UTC,view,5764655,1487580005411062629,,cnd,29.05,412120092,8adff31e-2051-4894-9758-224bfa8aec18
2019-12-01 00:00:02 UTC,cart,4958,1487580009471148064,,runail,1.19,494077766,c99a50e8-2fac-4c4d-89ec-41c05f114554
2019-12-01 00:00:05 UTC,view,5848413,1487580007675986893,,freedecor,0.79,348405118,722ffea5-73c0-4924-8e8f-371ff8031af4
2019-12-01 00:00:07 UTC,view,5824148,1487580005511725929,,,5.56,576005683,28172809-7e4a-45ce-bab0-5efa90117cd5
2019-12-01 00:00:09 UTC,view,5773361,1487580005134238553,,runail,2.62,560109803,38cf4ba1-4a0a-4c9e-b870-46685d105f95
2019-12-01 00:00:18 UTC,cart,5629988,1487580009311764506,,,1.19,579966747,1512be50-d0fd-4a92-bcd8-3ea3943f2a3b
Any help ?
Thanks
I am shifting my self-hosted parse to Amazon SNS and have followed all instructions on this link properly:
https://aws.amazon.com/blogs/mobile/migrating-from-parse-push-to-amazon-sns/
Now on step 4: Import Parse Push data when i run the java code
java -jar SNSImportTool.jar -s -f <PATH_TO_EXPORTED_PARSE_INSTALLATION_.JSON_FILE>
--apnsName <APNS_PLATFORM_APP_NAME_STEP_3.2.9>
--gcmName <GCM_PLATFORM_APP_NAME_STEP_3.2.9>
--topicName <SNS_TOPIC_NAME_STEP_3.2.10>
--awsaccess <AWS_KEY_ID_STEP_3.3>
--awssecret <SECRET_ACCESS_KEY_STEP_3.3>
i get the following output:
===========================================
Import from Parse to Amazon SNS
===========================================
Verify platform application: app/GCM/appname_MOBILEHUB_111111
Verify platform application: app/APNS/appname_MOBILEHUB_111111
Created 1 channel topics:SNS
Processing token APA91bEApdQ5PFqX77X53LZZevzw9ghwxuhvBwPu5E4Z1og1TTYTLFHv3oF5-AMszqZDfSBWCH4dn_zYK4Emx21j-30k6u0SGQ4KHbAWuuYbPCzSRm-9hvU
Could not create endpoint: Invalid parameter: PlatformApplicationArn Reason: Wrong number of slashes in relative portion of the ARN. (Service: AmazonSNS; Status Code: 400; Error Code: InvalidParameter; Request ID: 12b597de-1a46-5168-b1af-c661430db91f)
Processing token d806a7e6701df9250d8162e47b15ff78b857b285de38113d1b7f504ea92d46b5
Could not create endpoint: PlatformApplication does not exist (Service: AmazonSNS; Status Code: 404; Error Code: NotFound; Request ID: 419d4827-9238-5d46-a584-1998cb74e16e)
to clarify, for items in json that have deviceType as android i get the following error:
Invalid parameter: PlatformApplicationArn Reason: Wrong number of slashes in relative portion of the ARN
and deviceType as ios i get the following error:
Could not create endpoint: PlatformApplication does not exist
my json file looks like this:
{"results":[{ "channels" : ["SNS"], "_id" : "000f0KsszOH1", "_updated_at" : "2017-06-11T00:03:10.712+0000" , "UniqueId" : "79bb118ec2b71c3f", "appIdentifier" : "com.leApp", "parseVersion" : "1.8.0", "timeZone" : "Asia/Riyadh", "installationId" : "f05f3d8a-f045-35fe-a268-855eda3b2c78", "appVersion" : "2.2.8", "deviceType" : "ios", "_created_at" : "2015-12-15T12:49:23.954+0000", "pushType" : "gcm","deviceToken" : "APA99bEApdQ5PFqX77X53LYYevzw4ghwxuhvBwPu4C3Z1og1TTYTLFHv3oF5-AMszqZDfSBWCH4dn_zYK4Emx21j-30k6u0SGQ4KHbAWuuYbPCzSRm-9hvU", "localeIdentifier" : "en-GB" }]}
please advise how to fix these errors.
I configured WSO2 API Publisher (1.10.0) SAML SSO however login fails with the following error:
TID: [-1234] [] [2016-02-10 18:33:16,643] WARN {org.wso2.carbon.identity.sso.saml.processors.SPInitSSOAuthnRequestProcessor} - Destination validation for Authentication Request failed. Received: [null]. Expected one in the list: [https://identity.mydomain.pt:443/samlsso]
File publisher/site/conf/site.json:
"ssoConfiguration" : {
"enabled" : "true",
"issuer" : "apis-publisher",
"identityProviderURL" : "https://identity.mydomain.pt:443/samlsso",
"keyStorePassword" : "wso2carbon",
"identityAlias" : "wso2carbon",
"responseSigningEnabled":"true",
"keyStoreName" :"/home/wso2/wso2am-1.10.0/repository/resources/security/wso2carbon.jks",
//"nameIdPolicy" : "urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified", //If not specified, 'urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified' will be used
},
and the service provider configuration (sso-idp-config.xml):
<!-- API MANAGER PUBLISHER -->
<ServiceProvider>
<Issuer>apis-publisher</Issuer>
<AssertionConsumerServiceURLs>
<AssertionConsumerServiceURL>https://mgt.apis.mydomain.pt:443/publisher/jagg/jaggery_acs.jag</AssertionConsumerServiceURL>
<AssertionConsumerServiceURL>https://mgt.apis.mydomain.pt/publisher/jagg/jaggery_acs.jag</AssertionConsumerServiceURL>
</AssertionConsumerServiceURLs>
<DefaultAssertionConsumerServiceURL>https://mgt.apis.mydomain.pt:443/publisher/jagg/jaggery_acs.jag</DefaultAssertionConsumerServiceURL>
<EnableSingleLogout>true</EnableSingleLogout>
<SLOResponseURL/>
<SLORequestURL/>
<SAMLDefaultSigningAlgorithmURI>http://www.w3.org/2000/09/xmldsig#rsa-sha1</SAMLDefaultSigningAlgorithmURI>
<SAMLDefaultDigestAlgorithmURI>http://www.w3.org/2000/09/xmldsig#sha1</SAMLDefaultDigestAlgorithmURI>
<SignResponse>true</SignResponse>
<ValidateSignatures>true</ValidateSignatures>
<EncryptAssertion>false</EncryptAssertion>
<CertAlias>wso2carbon</CertAlias>
<EnableAttributeProfile>false</EnableAttributeProfile>
<IncludeAttributeByDefault>false</IncludeAttributeByDefault>
<ConsumingServiceIndex/>
<EnableAudienceRestriction>false</EnableAudienceRestriction>
<AudiencesList>
<Audience>apis-publisher</Audience>
</AudiencesList>
<EnableRecipients>false</EnableRecipients>
<RecipientList>
<Recipient/>
</RecipientList>
<EnableIdPInitiatedSSO>false</EnableIdPInitiatedSSO>
<EnableIdPInitSLO>false</EnableIdPInitSLO>
<ReturnToURLList>
<ReturnToURL/>
</ReturnToURLList>
</ServiceProvider>
I did the same configuration for the API Store and login is working.
I solve my problem by turning off signatures validation.
<ValidateSignatures>false</ValidateSignatures>
I have followed the instructions laid out in the documentation for GREG (5.1.0) to configure GREG to publish APIs to an external API Manager (1.10.0). I followed the configuration updates (modified the LifeCycle in the configuration.xml) and promoted a REST API. Unfortunately, when I promote the API to Production I don't receive any feedback in the GREG Publisher UI, the API is not imported into API Manager, and I receive the following errors in the GREG logs:
Note: I've scrubbed these logs for potentially sensitive information.
[2016-02-01 15:33:34,432] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - overview_name : xxxxxxxxxxxx
[2016-02-01 15:33:34,451] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - interface_transports : https
[2016-02-01 15:33:34,454] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - uritemplate_httpVerb : get
[2016-02-01 15:33:34,454] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - interface_swagger : /_system/governance/apimgt/applicationdata/api-docs/1.0.0/xxxxxxxxx
[2016-02-01 15:33:34,454] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - contacts_entry : Technical Owner: xxxxxxxxx
[2016-02-01 15:33:34,455] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - overview_endpointURL : http://xxxxxxxx/
[2016-02-01 15:33:34,456] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - uritemplate_authType : None
[2016-02-01 15:33:34,457] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - overview_description : xxxxxxxxx
[2016-02-01 15:33:34,457] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - overview_context : /xxxxxxxx/
[2016-02-01 15:33:34,457] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - overview_version : 0.0.4
[2016-02-01 15:33:34,460] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - uritemplate_urlPattern : /xxxxxx/{id}.xml
[2016-02-01 15:33:34,460] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - overview_provider : admin
[2016-02-01 15:33:34,460] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - endpoints_entry : Prod:https://xxxxxxxxxxxxxxx/xxxxxx/
[2016-02-01 15:33:34,460] ERROR {org.wso2.carbon.governance.registry.extensions.executors.apistore.RestServiceToAPIExecutor} - security_authenticationType : None
I have created a gist with the logs for both API Manager and GREG.
I have a log file which has bunch of lines, where each bunch is separated by blank lines. I want to grep certain lines (containing common pattern) from each bunch of line. Each bunch of line is about a mail. A sample log file is as follows:
#START#
03:48:19:798: : <23/08/2012 03:48:19:019>
03:48:19:798: : <---23/08/2012 03:48 --->
03:48:19:799: : MAIL FROM IP=1.2.3.4
03:48:19:799: : START CHECKING OF IPLIMIT
03:48:19:799: : STOP CHECKING OF IPLIMIT
03:48:20:848:In : MAIL FROM: <a#abc.com>
03:48:20:848: : [A:A:A]
03:48:20:849: : max attach size-->5242880
03:48:20:856: : User Is Authenticated with "a#abc.com and domain abc.com"
03:48:20:856: : Passed
03:48:20:987:In : RCPT TO: <x#xyz.com>
03:48:20:987: : email x#xyz.com
03:48:20:992: : [A:A:A]
03:48:20:999: : passed
03:48:20:999:Inside the Store Mails
03:48:20:999: : BCC feature is not applicable x#xyz.com
03:48:21:000: : BCC feature is not applicable from a#abc.com
03:48:21:000:Inside the Store
03:48:21:132:In : RCPT TO: <y#xyz.com>
03:48:21:132: : email y#xyz.com
03:48:21:133: : [A:A:A]
03:48:21:140: : passed
03:48:21:140:Inside the Store Mails
03:48:21:140: : BCC feature is not applicable y#xyz.com
03:48:21:140: : not authenticated
03:48:21:140:Inside the Store
03:48:21:271: : Data Received
03:50:32:049: : 552 Size Limit Exceeded(5242880)
03:50:32:049: : File Moved in LargeSize Folder....
03:50:32:049: : File Moved in LargeSize Folder....
03:50:32:049: : Connection closed
03:50:32:049: : File Deleted /home/Mail//mailbox/LargeSize/x#xyz.com:24085.444724474357(1345673901000)
03:50:32:051: : File Deleted /home/Mail//mailbox/LargeSize/y#xyz.com:39872.512978520455(1345673901140)
MAIL DATA : : 6815779 Bytes
Total: Conn : 16713 Quit By Host : 5565 Stored : 11134 Loop:0
#END#
W A R N I N G ---------------W A R N I N G
...Waiting for activity on port Total Thread Started & 16732 Stoped 16730
#START#
03:56:20:790: : <23/08/2012 03:56:20:020>
03:56:20:790: : <---23/08/2012 03:56 --->
03:56:20:791: : MAIL FROM IP=2.3.4.5
03:56:20:792: : IP IS FRIEND IN WHITELIST
03:56:20:834:In : MAIL FROM:<y#xyz.com>
03:56:20:834: : [A:A:A]
03:56:20:834: : null
03:56:20:834: : Passed
03:56:20:834:In : RCPT TO: <a#abc.com>
03:56:20:834: : email a#abc.com
03:56:20:835: : Mailing List
03:56:20:835: : [A:A:A]
03:56:20:836: : passed
03:56:20:836: : Proceesing maillist
03:56:20:839: : Data Received
03:56:20:865: : /home/Mail//mailbox/MailingList/a#abc.com:79602.39544573233(1345674380836) Msg Queued For Delivery
03:56:20:865: : Msg forward successfully
03:56:20:865: : /home/Mail//mailbox/MailingList/M14310.39892966699(1345674380837) Msg Queued For Delivery
MAIL DATA : : 27985 Bytes
Total: Conn : 16732 Quit By Host : 5582 Stored : 11135 Loop:0
#END#
...Waiting for activity on port Total Thread Started & 16735 Stoped 16731
#START#
03:56:23:957: : <23/08/2012 03:56:23:023>
03:56:23:957: : <---23/08/2012 03:56 --->
03:56:23:958: : MAIL FROM IP=2.3.4.5
03:56:23:959: : IP IS FRIEND IN WHITELIST
03:56:23:999:In : MAIL FROM: <x#xyz.com>
03:56:23:999: : [A:A:A]
03:56:23:999: : null
03:56:23:999: : Passed
03:56:23:999:In : RCPT TO: <y#xyz.com>
03:56:23:999: : email y#xyz.com
03:56:24:000: : [A:A:A]
03:56:24:007: : passed
03:56:24:008:Inside the Store Mails
03:56:24:009: : BCC feature is not applicable y#xyz.com
03:56:24:009: : not authenticated
03:56:24:009:Inside the Store
03:56:24:009: : Data Received
03:56:24:053: : /home/Mail//mailbox/External/y#xyz.com:50098.70335800691(1345674384009) Msg Queued For Delivery
03:56:24:054: : Msg forward successfully
MAIL DATA : : 28276 Bytes
Total: Conn : 16735 Quit By Host : 5582 Stored : 11136 Loop:0
#END#
Here, a#abc.com is an external mail id, and x#xyz.com, y#xyz.com are internal mail ids.
For each mail, the bunch of lines starting from #START# to #END# are generated.
From each bunch of lines I want to run some pattern matching. I only want those bunch of lines where mail is from an internal email id to external email id (second bunch of line).
I don't want bunch of lines where mail is from external email address/id to internal email id (1st bunch of line), or from an internal email id to internal email id. (3rd bunch of line).
And after I have the bunch of line where mail is from internal to external, I want to extract the line containing the word FROM and TO.
I tried using the RS, ORS, FS and OFS variables of awk to convert each bunch of line starting from and ending at #START# to make a single-line record, but couldn't. I could not replace the newlines by a separator such as | or ~. Also, I don't now how to run multiple pattern matching on each resource record.
I tried using /PATTERN/ option, but then could not run the grep command using system() function to get the lines to check the domain names. it gave me errors: sh: 1: not found. Could not break through it. I used the code:
if ($0 ~ /FROM/) { print $0 | system("egrep -i 'FROM|TO'") }
Also, if I try to export each record using following type of code, its not working:
for i in $(cat log_file | awk_file_givin_1_resource_record_at_a_time) ; do pattern_matching_commands ; done
It's no working cause the pattern matching is working on a line at a time, while I want it to work on the entire bunch at a time.
I think the following BASH script would work well, but you should benchmark it for the size of your logs:
#!/bin/bash
INTERNAL_DOMAINS="${1:-xyz.com|xyz.net}"
declare -i LINES BYTES VALIDS
LINES=0
BYTES=0
VALIDS=0
STATUS=stopped
while read LINE
do
if [ "$STATUS" = stopped ]
then
if [ "${LINE:0:7}" = "#START#" ]
then
STATUS=started
PARA=""
fi
else
if [ "${LINE:0:5}" = "#END#" ]
then
if [ $STATUS = valid ]
then
VALIDS+=1
echo "$PARA" | egrep -w "FROM|TO"
echo -e "$VALIDS matched\t----------------------------------------"
fi
STATUS=stopped
elif (echo "$LINE" | fgrep -q "RCPT TO") && (echo "$LINE" | egrep -qiv "#($INTERNAL_DOMAINS)")
then
STATUS=valid
PARA+="$LINE
"
else
PARA+="$LINE
"
fi
fi
LINES+=1
BYTES+=${#LINE}
BYTES+=1
echo -en "\rRead: lines: $LINES | kB: $(($BYTES/1024)) | matches: $VALIDS " >&2
done
You should set the above script as executable and run it like this to get progress output:
time ./filter.sh "one.int.com|two.int.com" < sample.log > report.out
If there is always a blank line between records, and never a blank line within a record, use awk's “paragraph mode”: set RS to the empty string.
awk -v RS= '
/^[0-9:]*In : MAIL FROM: <[^<>]*#example\.com>$/ &&
/^[0-9:]*In : RCPT TO: <[^<>]*#example\.com>$/ { … }
'
If you really need to use the #START# and #END# markers, accumulate data in variables as you go along. Do the processing then reset the variables when you reach #END#. Disable processing until the next #START# if necessary.
BEGIN { in_record = 1; }
/^#START#$/ { in_record = 1; }
!in_record { next; }
/^[0-9:]*In : MAIL FROM: <([^<>]*)>$/ { from = $0; sub(/.*</, "", from); sub(/>.*/, "", from); }
…
/^#END#$/ {
/* processing goes here */
from = "";
in_record = 0;
}