WSO2 API Manager Gateway cluster is not working - wso2

I configure the gateway cluster follow this guide:https://docs.wso2.com/display/AM210/Clustering+the+Gateway.
172.19.28.71: Gateway Manager Host
172.19.28.84: Gateway Worker Host
Gateway Manager axis2.xml:
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.am.internal.domain</parameter>
<parameter name="localMemberHost">172.19.28.71</parameter>
<parameter name="localMemberPort">4500</parameter>
<members>
<member>
<hostName>172.19.28.71</hostName>
<port>4500</port>
</member>
<member>
<hostName>172.19.28.84</hostName>
<port>4200</port>
</member>
</members>
Gateway Worker axis2.xml:
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.am.internal.domain</parameter>
<parameter name="localMemberHost">172.19.28.84</parameter>
<parameter name="localMemberPort">4200</parameter>
<members>
<member>
<hostName>172.19.28.71</hostName>
<port>4500</port>
</member>
<member>
<hostName>172.19.28.84</hostName>
<port>4200</port>
</member>
</members>
Publiser and Store api-manager.xml:
<Environment type="hybrid" api-console="true">
<Name>Internal</Name>
<Description>This is a hybrid gateway that handles both production and sandbox token traffic.</Description>
<ServerURL>https://172.19.28.71:9443/services/</ServerURL>
<Username>${admin.username}</Username>
<Password>${admin.password}</Password>
<GatewayEndpoint>http://172.19.28.84:8280,https://172.19.28.84:8243</GatewayEndpoint>
</Environment>
But When I test in API Console(Gateway Worker URL:https://172.19.28.84:8243/pizzashack/3.0.0), nothing response, following is Gateway Worker's log:
[2017-05-16 10:15:57,996] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = /pizzashack/3.0.0/menu {org.apache.synapse.mediators.builtin.LogMediator}
Gateway Manager URL (172.19.28.71:8243/pizzashack/3.0.0) is working.

You need to enable SVN based deployment synchronizer between gateways. Otherwise, pizzashack API won't be deployed in all the gateways.

Related

AWS Opensearch with SAML authentication

I have configured SAML for my AWS Opensearch Service Dashboard and keep getting 'Internal Server Error' after succesfully logging in to Okta and getting redirected to the sso endpoint (https://*****.eu-west-1.es.amazonaws.com/_dashboards/_opendistro/_security/saml/acs). I am using the service provider initiated login flow.
The SAML request looks like:
Host: *****
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: nl,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://***.okta.com/
Content-Type: application/x-www-form-urlencoded
Content-Length: 10427
Origin: https://***.okta.com
Connection: keep-alive
Cookie: security_authentication=Fe26.2**a179694d11de140222bccdb1b628732ad44371158089d49e851960cdfa74e711*rMo5VnNKA2FukJOaGT4zlw*9I-VJlFm20BlqKCAu7Sg9IUqtnLkPjVb-SBBMrEoSr9qX8NU24K6d7hiK6Q4ONPYo0cUbiGy25qudhs2DfYFrkRYTA1a0zf8fHRdxuQ6FNYXrkqWZ1s__kZVo-sAcwhcA6PbAXjFK3J-Mjy3-2N-VA**f25a0b1ddd9d36f949193a49ea74d88ff8fdb29fc2c0fc6d23102748a645a239*hL7oHPYT2TRQlaFw81ptxtKSFmXhzmcPkFkpF4U0j9U; STATE-TOKEN=fed6e87a-a743-4b36-a0e9-b62a579635a5
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: cross-site
HTTP/2.0 500 Internal Server Error
date: Mon, 18 Oct 2021 10:06:18 GMT
content-type: application/json; charset=utf-8
content-length: 77
x-amzn-requestid: 7c0a8527-c780-4bc9-b55a-4b8e0e468923
cache-control: private, no-cache, no-store, must-revalidate
osd-name: ip-10-212-37-230.eu-west-1.compute.internal
X-Firefox-Spdy: h2
<saml2p:Response Destination="****/_dashboards/_opendistro/_security/saml/acs"
ID="id12441206744048667167559313"
InResponseTo="ONELOGIN_e54a9652-c4b3-46b1-a9af-192b7892982e"
IssueInstant="2021-10-18T10:15:48.540Z"
Version="2.0"
xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
<saml2:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity"
xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"
>http://www.okta.com/exk4crh6pK2EwG3xz696</saml2:Issuer>
<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
<ds:SignedInfo>
<ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
<ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256" />
<ds:Reference URI="#id12441206744048667167559313">
<ds:Transforms>
<ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
<ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
<ec:InclusiveNamespaces PrefixList="xs"
xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#"
/>
</ds:Transform>
</ds:Transforms>
<ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256" />
<ds:DigestValue>wb2AqxWez2/KbOC81HYKxMoHDgxku2lXWXqrURo0k7k=</ds:DigestValue>
</ds:Reference>
</ds:SignedInfo>
<ds:SignatureValue>fSB69UWpOukV0hfX7gtoOd5lRU9Z7wKjWiYEfiAXi9eNLJGdzWA35eR5kxL/aSWp3r6TPj0ArcfgVOXgSKQfERWLsNaHuGFd2/vfEPsvb49NruitDgEmCVB+YMxTHZ3DujPlgf2/ADFI5hKV5nJfNkfFaJP/Y6cgnimDlBsXaV+E3wOrs2tfph5WbDYXIjKRlHb24cDJh7SRKK7WEmJR6HRPzlwCOkXGnc/UN1yqFHze+EMw+6buxPq04IoVA2waxNtsKwmm/LBSh5Up+UJdpvZ1ULF3GrTAbSiIbfxHHEQQWXTkwWJufdO+p24SOjcdgyMHqhtPO9Hs5Xa3lSISjg==</ds:SignatureValue>
<ds:KeyInfo>
<ds:X509Data>
<ds:X509Certificate>MIIDljCCAn6gAwIBAgIGAXyPqbX/MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYDVQQGEwJVUzETMBEG
A1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzENMAsGA1UECgwET2t0YTEU
MBIGA1UECwwLU1NPUHJvdmlkZXIxDDAKBgNVBAMMA3Z1YjEcMBoGCSqGSIb3DQEJARYNaW5mb0Bv
a3RhLmNvbTAeFw0yMTEwMTcxOTA5MzZaFw0zMTEwMTcxOTEwMzZaMIGLMQswCQYDVQQGEwJVUzET
MBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzENMAsGA1UECgwET2t0
YTEUMBIGA1UECwwLU1NPUHJvdmlkZXIxDDAKBgNVBAMMA3Z1YjEcMBoGCSqGSIb3DQEJARYNaW5m
b0Bva3RhLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJIbVHXi6RpEpOpM3JIj
0kZV4zw54qUVXNPbS8epUGAekHvpDqoCKGbWTG8vUBhLFQehhrBSzopLsVY2jzJIa+FKqviKWchv
5772IsVIn8/KAcTnW2lS8zs86mWbklL39/chIXT19m338E9goN0BhJEW0Hh1JmGzss4juE+WBhu0
7NTVVa2xSE5ELe1dyzwQrfbtn2HEBXe3PRnX1n7Mp8LGdUb3YQGQSzy90vc0D0rx24cCtnkaxWHn
c9vKj6U7unf1c+mJyihXVgGcRl89tc0CqYX41+TgH+Z6ygqNq+EKvx+94+4BvjHnDJMCLzKUAcS3
yiyyzTIra0Hj07hCux8CAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAhYATs51fcjglJZcob4nwFRvt
X8WWOxB3jyBweGgYdKKbkJuC8mmtINCaEMkCyPQNqxdEK0xRKqDZOD4Ay9s+1TKKn9L8FTCBIw9H
sYPBpjPLaU1+9Z1B47qHSXw/wgmSbVOCmr6ryLvj+rmaVEZRu38BS4E8EFUT1Bl8vGsKG2vbeT4j
1KUTNNh0v4N3Fe763EGvj0lbOLTWfTWu4/4cRcB5oZs6ybaZAfQ9DejYumFxf1dADxPqjjWEi4ay
gZZpQHNdRWd0mdBjFwnf9alxi7Kx69qqu6PRvQK0BZSI8teX+XnsihoV9D+QJPaTlXzjCqIUCRZD
hNwmmISRthgHsw==</ds:X509Certificate>
</ds:X509Data>
</ds:KeyInfo>
</ds:Signature>
<saml2p:Status xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol">
<saml2p:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success" />
</saml2p:Status>
<saml2:Assertion ID="id124412067441340881963049510"
IssueInstant="2021-10-18T10:15:48.540Z"
Version="2.0"
xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
<saml2:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity"
xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"
>http://www.okta.com/exk4crh6pK2EwG3xz696</saml2:Issuer>
<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
<ds:SignedInfo>
<ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
<ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256" />
<ds:Reference URI="#id124412067441340881963049510">
<ds:Transforms>
<ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
<ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
<ec:InclusiveNamespaces PrefixList="xs"
xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#"
/>
</ds:Transform>
</ds:Transforms>
<ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256" />
<ds:DigestValue>3XV/aXHpIRIXjJob312hhWsHbdoo5cqXCgoVM6MakEA=</ds:DigestValue>
</ds:Reference>
</ds:SignedInfo>
<ds:SignatureValue>TgN3QlXBHPumS7wrixp6R7oX30kyWWeT+dfu/raqBsqBOGb/iyvliPl3FX8AWkTRBjWou4Kbwfo6ashoUq0WvYNWwYZiEPwJVao8WPSzyHWLL8B0NCOoa68sQojWkVsTqGUQPDHqDq08Kxm0GZEudQuOf9SYwE4d+znmoUaBOorgZbFojbPD2AqnunAR9e9VCQYOsinoURVxrGqjIUnxwDpxvBcDl+i5CVcTCYmrG3VbPiLNaAdUXYAyyie4z3wa19reLk+O9NJ0EgqNxOnhEKc2SyJ7YxgA+UWTDjPIkqcww8AJl/LAmx6WY+KRu7nrlcwA4UWoNRuqgUaw2JoB7Q==</ds:SignatureValue>
<ds:KeyInfo>
<ds:X509Data>
<ds:X509Certificate>MIIDljCCAn6gAwIBAgIGAXyPqbX/MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYDVQQGEwJVUzETMBEG
A1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzENMAsGA1UECgwET2t0YTEU
MBIGA1UECwwLU1NPUHJvdmlkZXIxDDAKBgNVBAMMA3Z1YjEcMBoGCSqGSIb3DQEJARYNaW5mb0Bv
a3RhLmNvbTAeFw0yMTEwMTcxOTA5MzZaFw0zMTEwMTcxOTEwMzZaMIGLMQswCQYDVQQGEwJVUzET
MBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzENMAsGA1UECgwET2t0
YTEUMBIGA1UECwwLU1NPUHJvdmlkZXIxDDAKBgNVBAMMA3Z1YjEcMBoGCSqGSIb3DQEJARYNaW5m
b0Bva3RhLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJIbVHXi6RpEpOpM3JIj
0kZV4zw54qUVXNPbS8epUGAekHvpDqoCKGbWTG8vUBhLFQehhrBSzopLsVY2jzJIa+FKqviKWchv
5772IsVIn8/KAcTnW2lS8zs86mWbklL39/chIXT19m338E9goN0BhJEW0Hh1JmGzss4juE+WBhu0
7NTVVa2xSE5ELe1dyzwQrfbtn2HEBXe3PRnX1n7Mp8LGdUb3YQGQSzy90vc0D0rx24cCtnkaxWHn
c9vKj6U7unf1c+mJyihXVgGcRl89tc0CqYX41+TgH+Z6ygqNq+EKvx+94+4BvjHnDJMCLzKUAcS3
yiyyzTIra0Hj07hCux8CAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAhYATs51fcjglJZcob4nwFRvt
X8WWOxB3jyBweGgYdKKbkJuC8mmtINCaEMkCyPQNqxdEK0xRKqDZOD4Ay9s+1TKKn9L8FTCBIw9H
sYPBpjPLaU1+9Z1B47qHSXw/wgmSbVOCmr6ryLvj+rmaVEZRu38BS4E8EFUT1Bl8vGsKG2vbeT4j
1KUTNNh0v4N3Fe763EGvj0lbOLTWfTWu4/4cRcB5oZs6ybaZAfQ9DejYumFxf1dADxPqjjWEi4ay
gZZpQHNdRWd0mdBjFwnf9alxi7Kx69qqu6PRvQK0BZSI8teX+XnsihoV9D+QJPaTlXzjCqIUCRZD
hNwmmISRthgHsw==</ds:X509Certificate>
</ds:X509Data>
</ds:KeyInfo>
</ds:Signature>
<saml2:Subject xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion">
<saml2:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified">jeroenvanpelt#hotmail.com</saml2:NameID>
<saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
<saml2:SubjectConfirmationData InResponseTo="ONELOGIN_e54a9652-c4b3-46b1-a9af-192b7892982e"
NotOnOrAfter="2021-10-18T10:20:48.540Z"
Recipient="****/_dashboards/_opendistro/_security/saml/acs"
/>
</saml2:SubjectConfirmation>
</saml2:Subject>
<saml2:Conditions NotBefore="2021-10-18T10:10:48.540Z"
NotOnOrAfter="2021-10-18T10:20:48.540Z"
xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"
>
<saml2:AudienceRestriction>
<saml2:Audience>opensearch-saml</saml2:Audience>
</saml2:AudienceRestriction>
</saml2:Conditions>
<saml2:AuthnStatement AuthnInstant="2021-10-18T10:15:47.768Z"
SessionIndex="ONELOGIN_e54a9652-c4b3-46b1-a9af-192b7892982e"
xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"
>
<saml2:AuthnContext>
<saml2:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport</saml2:AuthnContextClassRef>
</saml2:AuthnContext>
</saml2:AuthnStatement>
<saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion">
<saml2:Attribute Name="roles"
NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"
>
<saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:type="xs:string"
>all_access</saml2:AttributeValue>
</saml2:Attribute>
</saml2:AttributeStatement>
</saml2:Assertion>
</saml2p:Response>
The error response in the browser is: {"statusCode":500,"error":"Internal Server Error","message":"Internal Error"}.
In the error logs (Cloudwatch), I have found the following messages:
[2021-10-18T01:45:40,286][WARN ][c.a.d.a.h.s.AuthTokenProcessorHandler] [ba6ca9920d4df640d8973f488f4c11c3] Error while validating SAML response in __PATH__
[2021-10-18T00:52:39,445][WARN ][r.suppressed ] [ba6ca9920d4df640d8973f488f4c11c3] path: __PATH__ params: {settings_filter=plugins.security.ssl.transport.pemkey_filepath,plugins.security.cert.oid,plugins.security.enable_snapshot_restore_privilege,plugins.security.audit.config.pemtrustedcas_filepath,reindex.ssl.supported_protocols,opendistro_security.compliance.history.external_config_enabled,plugins.security.ssl.transport.truststore_password,plugins.security.ssl.transport.keystore_alias,plugins.security.ssl.transport.keystore_type,plugins.security.check_snapshot_restore_write_privileges,plugins.security.advanced_modules_enabled,reindex.ssl.truststore.password,opendistro_security.*,plugins.security.ssl.transport.truststore_alias,plugins.security.unsupported.accept_invalid_config,plugins.security.audit.config.webhook.format,plugins.security.audit.config.webhook.ssl.pemtrustedcas_filepath,plugins.security.audit.config.pemkey_password,plugins.security.background_init_if_securityindex_not_exist,plugins.security.ssl.transport.enabled,plugins.security.audit.config.webhook.ssl.verify,plugins.security.ssl.transport.keystore_keypassword,plugins.security.protected_indices.roles,plugins.security.audit.config.index,plugins.security.ssl.http.keystore_alias,plugins.security.audit.config.webhook.url,plugins.security.allow_unsafe_democertificates,plugins.security.unsupported.restapi.allow_securityconfig_modification,plugins.security.allow_default_init_securityindex,plugins.security.ssl.http.truststore_type,plugins.security.ssl.transport.keystore_password,plugins.security.audit.config.log4j.logger_name,reindex.ssl.keystore.key_password,reindex.ssl.truststore.type,plugins.security.ssl.http.keystore_filepath,plugins.security.kerberos.krb5_filepath,plugins.security.ssl.transport.keystore_filepath,plugins.security.ssl.client.external_context_id,plugins.security.ssl.transport.pemcert_filepath,plugins.security.unsupported.inject_user.enabled,plugins.security.ssl.http.pemkey_password,opendistro_security.audit.enable_rest,reindex.ssl.key_passphrase,opendistro_security.audit.resolve_bulk_requests,plugins.security.restapi.password_validation_regex,plugins.security.unsupported.allow_now_in_dls,plugins.security.audit.config.type,plugins.security.ssl.transport.truststore_type,plugins.security.audit.threadpool.max_queue_len,plugins.security.audit.config.pemcert_filepath,plugins.security.audit.config.password,plugins.security.ssl.transport.enforce_hostname_verification,plugins.security.unsupported.restore.securityindex.enabled,plugins.security.*,plugins.security.config_index_name,plugins.security.audit.config.pemtrustedcas_content,plugins.security.ssl.transport.pemtrustedcas_filepath,reindex.ssl.truststore.path,plugins.security.ssl.http.pemcert_filepath,reindex.ssl.keystore.password,reindex.ssl.certificate_authorities,plugins.security.compliance.disable_anonymous_authentication,opendistro_security.audit.resolve_indices,plugins.security.audit.config.pemcert_content,plugins.security.ssl.http.truststore_password,plugins.security.ssl.http.crl.prefer_crlfile_over_ocsp,plugins.security.audit.config.pemkey_filepath,opendistro_security.compliance.history.read.metadata_only,opendistro_security.compliance.history.write.log_diffs,plugins.security.ssl.transport.extended_key_usage_enabled,plugins.security.unsupported.load_static_resources,plugins.security.compliance.salt,plugins.security.filter_securityindex_from_all_requests,reindex.ssl.certificate,plugins.security.ssl.http.crl.validate,reindex.ssl.verification_mode,opendistro_security.audit.enable_transport,plugins.security.ssl.http.crl.validation_date,plugins.security.audit.config.enable_ssl_client_auth,plugins.security.ssl.http.pemtrustedcas_filepath,plugins.security.ssl.http.keystore_keypassword,plugins.security.ssl_only,opendistro_security.compliance.history.write.metadata_only,opendistro_security.audit.log_request_body,plugins.security.unsupported.inject_user.admin.enabled,plugins.security.audit.config.webhook.ssl.pemtrustedcas_content,plugins.security.ssl.http.pemkey_filepath,plugins.security.ssl_cert_reload_enabled,plugins.security.audit.config.username,plugins.security.ssl.http.crl.disable_crldp,plugins.security.audit.threadpool.size,plugins.security.roles_mapping_resolution,plugins.security.audit.config.pemkey_content,reindex.ssl.keystore.path,plugins.security.ssl.http.enabled,plugins.security.kerberos.acceptor_keytab_filepath,plugins.security.system_indices.enabled,plugins.security.audit.config.cert_alias,reindex.ssl.client_authentication,reindex.ssl.keystore.type,plugins.security.audit.config.log4j.level,plugins.security.ssl.transport.truststore_filepath,plugins.security.audit.type,plugins.security.disabled,reindex.ssl.cipher_suites,plugins.security.disable_envvar_replacement,plugins.security.restapi.password_validation_error_message,plugins.security.ssl.http.crl.check_only_end_entities,opendistro_security.compliance.history.internal_config_enabled,opendistro_security.audit.exclude_sensitive_headers,secret_key,plugins.security.ssl.http.enable_openssl_if_available,plugins.security.ssl.http.clientauth_mode,plugins.security.protected_indices.enabled,plugins.security.unsupported.disable_rest_auth_initially,reindex.ssl.key,plugins.security.ssl.http.crl.file_path,plugins.security.audit.config.enable_ssl,plugins.security.kerberos.acceptor_principal,plugins.security.cert.intercluster_request_evaluator_class,reindex.ssl.keystore.algorithm,plugins.security.audit.config.verify_hostnames,plugins.security.ssl.http.keystore_type,plugins.security.ssl.http.truststore_filepath,plugins.security.cache.ttl_minutes,plugins.security.ssl.transport.pemkey_password,plugins.security.system_indices.indices,plugins.security.ssl.transport.enable_openssl_if_available,access_key,plugins.security.ssl.http.keystore_password,plugins.security.ssl.http.crl.disable_ocsp,plugins.security.ssl.http.truststore_alias,plugins.security.ssl.transport.principal_extractor_class,plugins.security.protected_indices.indices,plugins.security.ssl.transport.resolve_hostname,plugins.security.unsupported.disable_intertransport_auth_initially, filter_path=nodes.*.attributes.di_number}
OpenSearchSecurityException[OpenSearch Security not initialized for __PATH__]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:296)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:191)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97)
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:99)
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:88)
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:428)
at org.opensearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:717)
at org.opensearch.client.support.AbstractClient$ClusterAdmin.state(AbstractClient.java:747)
at org.opensearch.rest.action.admin.cluster.RestClusterStateAction.lambda$prepareRequest$0(RestClusterStateAction.java:125)
at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128)
at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:128)
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:271)
at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:353)
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:204)
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
__AMAZON_INTERNAL__
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
__AMAZON_INTERNAL__
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at __PATH__(Thread.java:834)
I started a new Opensearch cluster after users started to complain they could no longer log in to an older ES Cluster that was recently updated to Opensearch. Instead of SAML authentication, it was using Cognito authentication. As it was working before, and I followed the instructions (https://docs.aws.amazon.com/opensearch-service/latest/developerguide/saml.html) carefully again for both Cognito authentication and SAML authentication, it feels like something is wrong with Opensearch itself.
Check the SAML URLs configured in your IdP Broker as they changed from ES 7.x to OpenSearch 1.x. Also, try switching between using either the IdP or SP URL.
I've got exactly the same issue as you and I've managed to fix it by changing the SubjectKey to a custom SAML attribute:
I've added an attribute named email to my IdP configuration (Okta in your case, AWS SSO in my case).
Then I've set it up as the SubjectKey in AWS Opensearch. Here is my configuration for comparision:
SAML_METADATA=$(cat saml.xml | sed 's/"/\\\"/g' | sed ':a;N;$!ba;s/\n/\\n/g')
SAML_ENTITY_ID=$(grep entityID saml.xml | sed -r 's:.*entityID="(.*)".*:\1:')
aws opensearch update-domain-config \
--domain-name <name of the OpenSearch domain> \
--advanced-security-options '{"SAMLOptions":{"Enabled":true,"MasterUserName":"my-email#example.com", "Idp":{"EntityId":"'$SAML_ENTITY_ID'","MetadataContent":"'"$SAML_METADATA"'"}, "SessionTimeoutMinutes":180, "SubjectKey":"email"}}'
Here's what I did to solve the 500 internal server error with OpenSearch SSO login.
Note: Only if you are using AWS SSO
Log into AWS SSO console > Applications > Add New Application > search for OpenSearch (or select add custom SAML application)
Enter Display Name and Description and download AWS SSO SAML Metadata File
Go to the Application metadata section and add the Application ACS URL (Copy SSO URL (IdP initiated) from OpenSearch domain security configuration) > Save Changes
Go to Attribute mappings and add an attribute in the 1st column e.g email > value as ${user:email} > Save changes and Assignee required users.
Go to OpenSearch domain security configuration > upload the metadata file downloaded during step 2
Go to Additional Settings and add email (attribute name in step 4) to Subject key - optional
Go to your AWS SSO Start page you should see OpenSearch there.
Hope this helps

Cannot cluster WSO2 API manager 2.6.0 using Hazelcast

I am working on Hazelcast clustering WSO2 API Manager (following doc: https://docs.wso2.com/display/AM260/Working+with+Hazelcast+Clustering#WorkingwithHazelcastClustering-EnablingHazelcastclustering) Here I have two nodes under the same domain 127.0.0.1.
Let us assume A is running in port 4001 while B is running in port 4002. I have joined the two nodes as
A -
<members>
<member>
<hostName>127.0.0.1</hostName>
<port>4002</port>
</member>
<member>
<hostName>127.0.0.1</hostName>
<port>4001</port>
</member>
</members>
B -
<members>
<member>
<hostName>127.0.0.1</hostName>
<port>4001</port>
</member>
<member>
<hostName>127.0.0.1</hostName>
<port>4001</port>
</member>
</members>
I also tried as;
A -
<members>
<member>
<hostName>127.0.0.1</hostName>
<port>4002</port>
</member>
</members>
B -
<members>
<member>
<hostName>127.0.0.1</hostName>
<port>4002</port>
</member>
</members>
But both methods returned as below
2019-11-27 13:23:16,763] INFO - SocketAcceptor [127.0.0.1]:0 [wso2.am.domain] [3.5.4] Accepting socket connection from /127.0.0.1:51206
[2019-11-27 13:23:16,763] INFO - TcpIpConnectionManager [127.0.0.1]:0 [wso2.am.domain] [3.5.4] Established socket connection between /127.0.0.1:4001
[2019-11-27 13:23:16,764] WARN - TcpIpConnectionManager [127.0.0.1]:0 [wso2.am.domain] [3.5.4] Wrong bind request from Address[127.0.0.1]:0! This node is not requested endpoint: Address[127.0.0.1]:4001
[2019-11-27 13:23:16,764] INFO - TcpIpConnection [127.0.0.1]:0 [wso2.am.domain] [3.5.4] Connection [/127.0.0.1:51206] lost. Reason: Socket explicitly closed
[2019-11-27 13:23:44,354] INFO - TcpIpConnectionManager [127.0.0.1]:0 [wso2.am.domain] [3.5.4] Established socket connection between /127.0.0.1:51211
[2019-11-27 13:23:44,359] INFO - TcpIpConnection [127.0.0.1]:0 [wso2.am.domain] [3.5.4] Connection [Address[127.0.0.1]:4002] lost. Reason: java.io.EOFException[Remote socket closed!]
[2019-11-27 13:23:44,360] WARN - ReadHandler [127.0.0.1]:0 [wso2.am.domain] [3.5.4] hz.wso2.am.domain.instance.IO.thread-in-1 Closing socket to endpoint Address[127.0.0.1]:4002, Cause:java.io.EOFException: Remote socket closed!
How to solve this issue?
[Issue solved]
The answer is "Use local machine IP instead of localhost ". There are 02 options.
Option 01
Add the following to the tasks/main.yml
# Get local IP
- name: get local ip
debug:
var: ansible_default_ipv4.address
and this to site.yml
- hosts: localhost
connection: local
Option 02
Add the following snippet to site.yml
---
- hosts: localhost
roles:
- carbon
connection: local
tasks:
- debug: var=ansible_all_ipv4_addresses
- debug: var=ansible_default_ipv4.address

WSO2 SP upgrading from DAS

I am trying to upgrade from WSO2 DAS to SP; which involves rewriting the event flow definition with Siddhi SQL script.
my object is that all changes be restricted within Siddhi SQL script(thus within SP).
I am using DAS as a simple "Message Broker" in a micro service context.
as shown in the diagram:
there are three(3) micro services: A, B, C; service A calls service B, and C.
service A issues an "wso2event" with the wso2 data agent;
the event is received by a receiver (type="wso2event") in the DAS;
two(2) publishers (eventAdapterType="soap") were used to form two SOAP messages and published respectively to service B and C
below are my artifacts definitions:
stream:
{
"name": "ip3c",
"version": "1.0.0",
"nickName": "ip3c.receiverservice.senderservice.follow",
"description": "follow event from receiver to sender and userevent",
"metaData": [
{
"name": "sender",
"type": "STRING"
}
],
"correlationData": [
{
"name": "host",
"type": "STRING"
}
],
"payloadData": [
{
"name": "message",
"type": "STRING"
}
]
}
receiver:
<?xml version="1.0" encoding="UTF-8"?>
<!-- gbb2.receiverservice.follow -->
<eventReceiver name="gbb2" statistics="disable"
trace="disable" xmlns="http://wso2.org/carbon/eventreceiver">
<from eventAdapterType="wso2event">
<property name="receiving.events.duplicated.in.cluster">false</property>
</from>
<mapping customMapping="disable" type="wso2event"/>
<to streamName="ip3c" version="1.0.0"/>
</eventReceiver>
publisher 1:
<?xml version="1.0" encoding="UTF-8"?>
<!-- pnbu.senderservice.follow -->
<eventPublisher name="pnbu" statistics="disable"
trace="disable" xmlns="http://wso2.org/carbon/eventpublisher">
<from streamName="ip3c" version="1.0.0"/>
<mapping customMapping="enable" type="xml">
<inline>
<sen:follow xmlns:sen="http://gubnoi.com/SenderService/">
<sender>{{meta_sender}}</sender>
<host>{{correlation_host}}</host>
<message>{{message}}</message>
</sen:follow>
</inline>
</mapping>
<to eventAdapterType="soap">
<property name="soapHeaders">SOAPAction: http://gubnoi.com/SenderService/follow</property>
<property name="url">http://sender.gubnoi.com:10102/services/SenderService/</property>
</to>
</eventPublisher>
publisher 2:
<?xml version="1.0" encoding="UTF-8"?>
<!-- tvzh.userevent.receiverfollow -->
<eventPublisher name="tvzh" statistics="disable"
trace="disable" xmlns="http://wso2.org/carbon/eventpublisher">
<from streamName="ip3c" version="1.0.0"/>
<mapping customMapping="enable" type="xml">
<inline>
<use:save xmlns:use="http://gubnoi.com/UserEvents/">
<title>u5khurw3</title>
<obj>{{meta_sender}}</obj>
<host>{{correlation_host}}</host>
<bucket>{{message}}</bucket>
</use:save>
</inline>
</mapping>
<to eventAdapterType="soap">
<property name="soapHeaders">SOAPAction: http://gubnoi.com/UserEvents/save</property>
<property name="url">http://userevents.gubnoi.com:10304/services/UserEvents/</property>
</to>
</eventPublisher>
I searched around, and could not find any useful examples.
Can anyone please give any help
thanks
WSO2 Stream Processor has a documentation on upgrading from a previous release. You can refer the documentation to create siddhi artifact from your streams, receivers and publishers.

Redirect http requests to https with tomcat server and aws load balancer

I have set up an ec2 instance with a java web application running on a tomcat server and an aws application load balancer which directs port 80 and 443 to port 8080 on the ec2 instance.
I have changed the server.xml configuration on tomcat to the following
<Connector port="8080" protocol="HTTP/1.1"
proxyPort="443"
proxyName="sub.mydomain.com"
scheme="https"
secure="true"
connectionTimeout="20000"
redirectPort="8443" />
Now I am able to connect go to "sub.mydomain.com" and "https ://sub.mydomain.com". But how do I redirect all "http ://sub.mydomain.com" requests to "https ://sub.mydomain.com"?
Just add this in your web.xml
<security-constraint>
<web-resource-collection>
<web-resource-name>HTTPSOnly</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>

wso2 server redeployment error "Faulty Carbon Applications."

I am getting an error whenever I tried to redeploy the app to the esb. I am trying to merge the 2 response from different api and then map and manipulate that response. I am also using the latest 5.0.0 BETA version both esb tooling and esb.
Here is the fault reason
org.apache.axis2.deployment.DeploymentException: API deployment from
the file :
/Users/me/Downloads/wso2esb-5.0.0-BETA2/tmp/carbonapps/-1234/1468999438631movieapiApp_1.0.0.car/movieAPI_1.0.0/movieAPI-1.0.0.xml
: Failed. at
org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:213)
at
org.wso2.carbon.application.deployer.synapse.SynapseAppDeployer.deployArtifacts(SynapseAppDeployer.java:131)
at
org.wso2.carbon.application.deployer.internal.ApplicationManager.deployCarbonApp(ApplicationManager.java:263)
at
org.wso2.carbon.application.deployer.CappAxis2Deployer.deploy(CappAxis2Deployer.java:72)
at
org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
at
org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:807)
at
org.apache.axis2.deployment.repository.util.WSInfoList.update(WSInfoList.java:144)
at
org.apache.axis2.deployment.RepositoryListener.update(RepositoryListener.java:377)
at
org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:254)
at
org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371)
at
org.apache.axis2.deployment.scheduler.SchedulerTask.checkRepository(SchedulerTask.java:59)
at
org.apache.axis2.deployment.scheduler.SchedulerTask.run(SchedulerTask.java:67)
at
org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.runAxisDeployment(CarbonDeploymentSchedulerTask.java:93)
at
org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.axis2.deployment.DeploymentException: API deployment from
the file :
/Users/me/Downloads/wso2esb-5.0.0-BETA2/tmp/carbonapps/-1234/1468999438631movieapiApp_1.0.0.car/movieAPI_1.0.0/movieAPI-1.0.0.xml
: Failed. at
org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:201)
... 20 more Caused by:
org.apache.synapse.deployers.SynapseArtifactDeploymentException: API
deployment from the file :
/Users/me/Downloads/wso2esb-5.0.0-BETA2/tmp/carbonapps/-1234/1468999438631movieapiApp_1.0.0.car/movieAPI_1.0.0/movieAPI-1.0.0.xml
: Failed. at
org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.handleSynapseArtifactDeploymentError(AbstractSynapseArtifactDeployer.java:474)
at
org.apache.synapse.deployers.APIDeployer.deploySynapseArtifact(APIDeployer.java:71)
at
org.wso2.carbon.rest.api.ApiDeployer.deploySynapseArtifact(ApiDeployer.java:34)
at
org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:194)
... 20 more Caused by: org.apache.synapse.SynapseException: Duplicate
resource definition by the name: movieapi at
org.apache.synapse.config.SynapseConfiguration.handleException(SynapseConfiguration.java:1627)
at
org.apache.synapse.config.SynapseConfiguration.addAPI(SynapseConfiguration.java:414)
at
org.apache.synapse.deployers.APIDeployer.deploySynapseArtifact(APIDeployer.java:59)
... 22 more
and here's my api
<?xml version="1.0" encoding="UTF-8"?>
<api context="/movieapi" name="movieapi" xmlns="http://ws.apache.org/ns/synapse">
<resource methods="GET" uri-template="/*">
<inSequence>
<property name="ROOT" scope="default">
<root:movie xmlns:root="www.wso2esb.com"/>
</property>
<log level="full"/>
<clone continueParent="true" id="movie" sequential="true">
<target>
<sequence>
<send>
<endpoint>
<address format="rest" uri="https://api.themoviedb.org/3/movie/tt0918940?api_key=code&append_to_response=casts,images%22"/>
</endpoint>
</send>
</sequence>
</target>
<target>
<sequence>
<send>
<endpoint>
<address format="rest" uri="https://www.omdbapi.com/?type=movie&i=tt0918940"/>
</endpoint>
</send>
</sequence>
</target>
</clone>
</inSequence>
<outSequence>
<aggregate id="movie">
<completeCondition>
<messageCount max="-1" min="-1"/>
</completeCondition>
<onComplete enclosingElementProperty="ROOT" expression="//jsonObject" xmlns:dummy="http://org.dummy" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<datamapper config="gov:datamapper/movieapiMapping.dmc" inputSchema="gov:datamapper/movieapiMapping_inputSchema.json" inputType="JSON" outputSchema="gov:datamapper/movieapiMapping_outputSchema.json" outputType="JSON"/>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<respond/>
</onComplete>
</aggregate>
</outSequence>
<faultSequence/>
</resource>
</api>
and this is from the terminal
[2016-07-20 15:36:48,890] ERROR - DataMapperMediator DataMapper mediator : mapping configuration is null
The error you are getting in the terminal DataMapperMediator DataMapper mediator : mapping configuration is null is because the relevant registry resources are not deployed (more precisely, the configuration .dmc file). When you are deploying the capp, make sure that all the registry resources are included as well. Also when the capp deployment fails, can you check in the management console for any faulty services/APIs?
You get the error indicating Data mapper configurations null. So you have to add relevant configurations as you given in the proxy:
<datamapper config="gov:datamapper/movieapiMapping.dmc" inputSchema="gov:datamapper/movieapiMapping_inputSchema.json" inputType="JSON" outputSchema="gov:datamapper/movieapiMapping_outputSchema.json" outputType="JSON"/>
For example: You indicated about gov:datamapper/movieapiMapping.dmc
This should be available in governance registry.Please deploy relevant configs in to ESB.
Open the pom.xml file inside your CompositeApplication.
Edit it in the Design mode.
Under dependencies, you can see the artifacts which you can include.
Tick the checkbox near your relevant Registry artifact. Save pom file.
Now redeploy the application to the server or Regenerate your car file