Tensorflow/Tensorboard on s3 - amazon-web-services

We log the tensorboard logs of our training process to s3 and meanwhile I would like to have a look at it. Therefore I'm executing the following command:
tensorboard --logdir=s3://mlflow/1/7327710663534420b09581b9fa601bfc/artifacts/tensorboard_logs/ --bind_all
I know that that directory exists because I checked with the s3cmd tool.
My env variables which I exported earlier:
AWS_ACCESS_KEY_ID="<MyKey>"
AWS_SECRET_ACCESS_KEY="<MySecret>"
S3_ENDPOINT="<MyCustomEndpoint>"
S3_USE_HTTPS="1"
S3_VERIFY_SSL="0"
And some logs:
2020-01-31 16:43:54.140206: I tensorflow/core/platform/s3/aws_logging.cc:54] Initializing config loader against fileName /Users/taawech0//.aws/config and using profilePrefix = 1
2020-01-31 16:43:54.140242: I tensorflow/core/platform/s3/aws_logging.cc:54] Initializing config loader against fileName /Users/taawech0//.aws/credentials and using profilePrefix = 0
2020-01-31 16:43:54.140250: I tensorflow/core/platform/s3/aws_logging.cc:54] Setting provider to read credentials from /Users/taawech0//.aws/credentials for credentials file and /Users/taawech0//.aws/config for the config file , for use with profile default
2020-01-31 16:43:54.140278: I tensorflow/core/platform/s3/aws_logging.cc:54] Creating AWSHttpResourceClient with max connections2 and scheme http
2020-01-31 16:43:54.140289: I tensorflow/core/platform/s3/aws_logging.cc:54] Initializing CurlHandleContainer with size 2
2020-01-31 16:43:54.140299: I tensorflow/core/platform/s3/aws_logging.cc:54] Creating Instance with default EC2MetadataClient and refresh rate 300000
2020-01-31 16:43:54.140303: I tensorflow/core/platform/s3/aws_logging.cc:54] Added EC2 metadata service credentials provider to the provider chain.
2020-01-31 16:43:54.140316: I tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2020-01-31 16:43:54.140364: I tensorflow/core/platform/s3/aws_logging.cc:54] Initializing CurlHandleContainer with size 25
2020-01-31 16:43:54.140418: I tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2020-01-31 16:43:54.140528: I tensorflow/core/platform/s3/aws_logging.cc:54] Pool grown by 2
2020-01-31 16:43:54.140534: I tensorflow/core/platform/s3/aws_logging.cc:54] Connection has been released. Continuing.
2020-01-31 16:43:54.270240: E tensorflow/core/platform/s3/aws_logging.cc:60] HTTP response code: 404
Exception name:
Error message: No response body.
5 response headers:
cache-control : must-revalidate,no-cache,no-store
content-type : text/html;charset=iso-8859-1
date : Fri, 31 Jan 2020 15:43:57 GMT
x-amz-id-2 :
x-amz-request-id : 8abebd2a:16ff02dd20c:3755:3dc
2020-01-31 16:43:54.270288: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2020-01-31 16:43:54.270354: I tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2020-01-31 16:43:54.270446: I tensorflow/core/platform/s3/aws_logging.cc:54] Connection has been released. Continuing.
TensorBoard 2.1.0 at http://UM01549:6006/ (Press CTRL+C to quit)
2020-01-31 16:43:54.406342: I tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2020-01-31 16:43:54.406515: I tensorflow/core/platform/s3/aws_logging.cc:54] Connection has been released. Continuing.
2020-01-31 16:43:54.448773: E tensorflow/core/platform/s3/aws_logging.cc:60] HTTP response code: 404
Exception name:
Error message: No response body.
5 response headers:
cache-control : must-revalidate,no-cache,no-store
content-type : text/html;charset=iso-8859-1
date : Fri, 31 Jan 2020 15:43:58 GMT
x-amz-id-2 :
x-amz-request-id : 8abebd2d:16ff0369a72:371b:722
2020-01-31 16:43:54.448829: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
At that point I'm like really confused. Thanks for your help.

Related

Jenkins EC2 plugin: instances not starting

This is my Jenkins EC2 configuration:
URL: $JENKINS_URL/configureClouds/
Add new cloud: Amazon EC2
Name: Amazon EC2 eu-central-1
Amazon EC2 Credentials: AKIA...
Region: eu-central-1
EC2 Key Pair's Private Key: ubuntu
Test connection: success
Advanced...
Instance Cap: 3
No delay provisioning: checked
Add AMI
Description: Linux node
AMI ID: ami-0293...
Check AMI: 05052029...
Instance Type: T3aMedium
EBS Optimized: checked
Monitoring: checked
T2 Unlimited: checked
Security group names: sg-0c2d... (opens SSH port 22)
Remote FS root: ./jenkins
Remote user: ubuntu
AMI Type: unix
Labels: aws ubuntu linux
Usage: Use this node as much as possible
Idle termination time: 30
Advanced...
Number of executors: 2
Stop/Disconnect on Idle Timeout: checked
Minimum number of instances: 1
Minimum number of spare instances: 0
Instance cap: 10
Block device mapping: /dev/sda1=snap-0eadbe3f...:200:true:gp2, /dev/sdb=ephemeral0, /dev/sdc=ephemeral1
Associate Public IP: checked
Connection Strategy: Public DNS
Host Key Verification Strategy: off
Maximum Total Uses: 10
Environment variables: checked
(not listing all environment variables)
Tool locations: checked
(not listing all tool locations)
With this configuration, I would expect that at least 1 EC2 instance would be started, but no instance is started.
In the nodes page in Jenkins when I hit the provision via button, I get an error:
Oops! A problem occurred while processing the request. Logging ID=8ead3651-3809-4a47-984c-e0e494c705bb
In /log/all I have:
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate getImage
Getting image for request {ExecutableUsers: [],Filters: [],ImageIds: [ami-0293c4ed***],Owners: []}
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{description='Linux node', labels='aws ubuntu linux'}. Considering launching
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice
AMI had /dev/sda1
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice
{DeleteOnTermination: true,SnapshotId: snap-0eadbe3f***,VolumeSize: 20,VolumeType: gp2,Encrypted: false}
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{description='Linux node', labels='aws ubuntu linux'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Stop
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{description='Linux node', labels='aws ubuntu linux'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-0293c4ed***]}, {Name: instance-type,Values: [t3a.medium]}, {Name: key-name,Values: [***]}, {Name: tag:jenkins_server_url,Values: [https://jenkins.***.com/]}, {Name: tag:jenkins_slave_type,Values: [demand_Linux node]}],InstanceIds: [],}
Apr 14, 2021 5:34:37 PM WARNING hudson.init.impl.InstallUncaughtExceptionHandler handleException
Caught unhandled exception with ID c080ae42-6b7b-47aa-93ea-1b8064503c1c
com.amazonaws.services.ec2.model.AmazonEC2Exception: Value () for parameter groupId is invalid. The value cannot be empty (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: da74dbbf-0685-45ac-8454-c3f5d1b4c700; Proxy: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:29240)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:29207)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:29196)
at com.amazonaws.services.ec2.AmazonEC2Client.executeRunInstances(AmazonEC2Client.java:28011)
at com.amazonaws.services.ec2.AmazonEC2Client.runInstances(AmazonEC2Client.java:27980)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:1100)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:1042)
at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:867)
at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:693)
at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:430)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:111)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at org.jenkinsci.plugins.modernstatus.ModernStatusFilter.doFilter(ModernStatusFilter.java:50)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.lang.Thread.run(Thread.java:748)
Write up of the comments for anyone else looking for help diagnosing EC2 Agent Plugin issue.
When you have configured your agents go to the Nodes page (Jenkins URL/computer)
Hit the button to Provision a new agent from your cloud
If there is a configuration issue you will get Evil Jenkins and a Logging ID
Go to Jenkins logs page (Jenkins URL/log/all) and search that ID
This should give you the stack trace from the AWS SDK call which will help you to narrow down whether is it missing config or IAM permissions etc at fault
If there were no config errors you would get taken to the node that is being launched config page where you would be able to see its EC2 startup log to check for any User Data or AMI issues.
For this particular issue 'Value () for parameter groupId is invalid' the plugin is expecting you to provide a list or at least one subnet-id in configuration.
I faced the same issue and was resolved with this.

To change the status code from 200 to 404 when stepfunction not found

I'm trying to Execute the AWS step function from API Gateway, It's working as expected.
Whenever I'm passing the input, statemachinearn(stepfunction name to execute) It's triggering the step function.
But It's still returning the status code 200, whenever it's not able to find the stepfunction, I want to return the status code 404 if the apigateway not found that stepfunction.
Could you please help me on that
Response:
Status: 200ok
Expected:
Status: 404
Thanks,
Harika.
As per the documentation StartExecution API call do return 400 Bad Request for non existent statemachine which is correct as RESTful API standard.
StateMachineDoesNotExist
The specified state machine does not exist.
HTTP Status Code: 400
From the RESTful API point of view, endpoint /execution/(which I created in API Gateway for the integration setup) is a resource, no matter it accepts GET or POST or something else. 404 is only appropriate when the resource /execution/ itself does not exist. If /execution/ endpoint exists, but its invocation failed (no matter what the reasons), the response status code must be something other than 404.
So in the case of the returned response(200) for POST call with non-existent statemachine it is correct. But when API Gateway tried to make the call to non-existent statemachine it got 404 from StartExecution api call which it eventually wrapped into a proper message instead of returning 404 http response.
curl -s -X POST -d '{"input": "{}","name": "MyExecution17","stateMachineArn": "arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine"}' https://123456asdasdas.execute-api.eu-central-1.amazonaws.com/v1/execution|jq .
{
"__type": "com.amazonaws.swf.service.v2.model#StateMachineDoesNotExist",
"message": "State Machine Does Not Exist: 'arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine1'"
}
Let's say you create another MethodResponse where you can provide an exact HTTP Status Code in your case 404 which you want to return and you do an Integration Response where you have to choose the Method Response by providing either Exact HTTP Responce Code(400 -> Upstream response from the **StartExecution** API Call) OR a Regex -> (4\{d}2) matching all the 4xx errors.
In that case you will be giving 404 for all the responses where the upstream error 4xx StartExecution Errors
ExecutionAlreadyExists -> 400
ExecutionLimitExceeded -> 400
InvalidArn -> 400
InvalidExecutionInput -> 400
InvalidName -> 400
StateMachineDeleting -> 400
StateMachineDoesNotExist -> 400
Non Existent State Machine:
curl -s -X POST -d '{"input": "{}","name": "MyExecution17","stateMachineArn": "arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine1"}' https://123456asdasdas.execute-api.eu-central-1.amazonaws.com/v1/execution|jq .
< HTTP/2 404
< date: Sat, 30 Jan 2021 14:12:16 GMT
< content-type: application/json
...
{
"__type": "com.amazonaws.swf.service.v2.model#StateMachineDoesNotExist",
"message": "State Machine Does Not Exist: 'arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine1'"
}
Execution Already Exists
curl -s -X POST -d '{"input": "{}","name": "MyExecution17","stateMachineArn": "arn:aws:states:eu-central-1:1234567890:stateMachine:mystatemachine"}' https://123456asdasdas.execute-api.eu-central-1.amazonaws.com/v1/execution|jq .
* We are completely uploaded and fine
< HTTP/2 404
< date: Sat, 30 Jan 2021 14:28:27 GMT
< content-type: application/json
{
"__type": "com.amazonaws.swf.service.v2.model#ExecutionAlreadyExists",
"message": "Execution Already Exists: 'arn:aws:states:eu-central-1:1234567890:execution:mystatemachine:MyExecution17'"
}
Which I think will be misleading.

GoogleStorageException - 401 Unauthorized / Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket

I want to transfer data from GCS to BigQuery by embulk and digdag.
But error occurs.
com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized
.......
Error: org.embulk.config.ConfigException: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
↓ Details
command :
embulk run XXXX.yaml
XXXX.yaml :
in:
type: gcs
bucket: <bucket name>
path_prefix: <file path>
auth_method: compute_engine
parser:
type: poi_excel
sheets: <sheet name>
skip_header_lines: 4
columns:
- {name: 'name', type: string}
.
.
.
out:
type: bigquery
mode: replace
project: <project name>
dataset: <dataset name>
table: <table name>
auth_method: compute_engine
schema_file: <file name of json type>
gcs_bucket: <gcs tmp bucket name>
output :
$ embulk run target_item_bottoms_config.yaml
2020-07-22 14:27:36.559 +0900: Embulk v0.9.23
2020-07-22 14:27:37.609 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2020-07-22 14:27:40.577 +0900 [INFO] (main): Gem's home and path are set by default: "/Users/oniki/.embulk/lib/gems"
2020-07-22 14:27:41.662 +0900 [INFO] (main): Started Embulk v0.9.23
2020-07-22 14:27:41.853 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-gcs (0.3.2)
2020-07-22 14:27:46.263 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.6.4)
2020-07-22 14:27:46.369 +0900 [INFO] (0001:transaction): Loaded plugin embulk-parser-poi_excel (0.1.7)
org.embulk.exec.PartialExecutionException: org.embulk.config.ConfigException: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:566)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
at org.embulk.spi.Exec.doWith(Exec.java:22)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:291)
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:155)
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
at org.embulk.cli.Main.main(Main.java:64)
Suppressed: java.lang.NullPointerException
at org.embulk.exec.BulkLoader.doCleanup(BulkLoader.java:463)
at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:397)
at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:394)
at org.embulk.spi.Exec.doWith(Exec.java:22)
at org.embulk.exec.BulkLoader.cleanup(BulkLoader.java:394)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:245)
... 5 more
Caused by: org.embulk.config.ConfigException: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
at org.embulk.input.gcs.AuthUtils.newClient(AuthUtils.java:81)
at org.embulk.input.gcs.GcsFileInput.listFiles(GcsFileInput.java:49)
at org.embulk.input.gcs.GcsFileInputPlugin.transaction(GcsFileInputPlugin.java:59)
at org.embulk.spi.FileInputRunner.transaction(FileInputRunner.java:62)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:507)
... 11 more
Caused by: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:226)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:366)
at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:338)
at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:335)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.StorageImpl.listBlobs(StorageImpl.java:334)
at com.google.cloud.storage.StorageImpl.list(StorageImpl.java:290)
at org.embulk.input.gcs.AuthUtils.newClient(AuthUtils.java:77)
... 15 more
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized
{
"code" : 401,
"errors" : [ {
"domain" : "global",
"location" : "Authorization",
"locationType" : "header",
"message" : "Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.",
"reason" : "required"
} ],
"message" : "Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:401)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1097)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:499)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:356)
... 23 more
Error: org.embulk.config.ConfigException: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
my environment :
$ gcloud config list
[compute]
region = us-east1
zone = us-east1-c
[core]
account = myname#xxx.com
disable_usage_reporting = False
project = <project ID>
Your active configuration is: [default]
$ gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* myname#xxxx.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
$ gsutil ls
gs://<bucket name>
my gcp IAM role :
owner
I understand that the solution to this error is authorization.
But my preferences seem to be fine.
what's wrong?
As the documentation [1], if we have 401- Unauthorized error then there could be many reasons, please have a related list of reasons listed below [followed the link 1], which could be helpful for troubleshooting:
Reason:AuthenticationRequiredRequesterPays
Access to a Requester Pays bucket requires authentication.
Reason: authError
This error indicates a problem with the authorization provided in the request to Cloud Storage. The following are some situations where that will occur:
The OAuth access token has expired and needs to be refreshed. This can be avoided by refreshing the access token early, but code can also catch this error, refresh the token and retry automatically.
Multiple non-matching authorizations were provided; choose one mode only.
The OAuth access token's bound project does not match the project associated with the provided developer key.
The Authorization header was of an unrecognized format or uses an unsupported credential type.
reason:lockedDomainExpired
When downloading content from a cookie-authenticated site, e.g., using the Storage Browser, the response will redirect to a temporary domain. This error will occur if access to said domain occurs after the domain expires. Issue the original request again, and receive a new redirect.
Reason: push.webhookUrlUnauthorized
Requests to storage.objects.watchAll will fail unless you verify you own the domain.
Reason: required
Access to a non-public method that requires authorization was made, but none was provided in the Authorization header or through other means.
[1] https://cloud.google.com/storage/docs/json_api/v1/status-codes#401_Unauthorized
I try locally , and create Service Account Key and save at local .
◾️XXXX.yaml
before
auth_method: compute_engine
after
auth_method: json_key
json_keyfile: /path/to/json_keyfile.json

Getting "SignatureDoesNotMatch" error with delete Bucket Replication operation in Amazon s3

I'm trying to use delete bucket replication operation in Amazon s3 using rest API [1]. For this, I have implemented the AWS signature calculation class[2] and a method to call the deleteBucketReplication rest API [3]. But I'm getting 403 "SignatureDoesNotMatch" error. Please find the request sending to backed here [4].
Can anyone help me to overcome this issue?
[1] https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketDELETEreplication.html
[2] https://github.com/Biruntha/AWS_S3_Example/blob/master/src/main/java/AWSSignatureV4Generator.java
[3]https://github.com/Biruntha/AWS_S3_Example/blob/master/src/main/java/AWSV4AuthTest.java#L13
[4]
Canonical Request: DELETE
/?replication
host:testbuckkkbiru.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:Tue, 09 Oct 2018 05:55:08 GMT
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
String to sign: AWS4-HMAC-SHA256
Tue, 09 Oct 2018 05:55:08 GMT
20181009/us-east-2/s3/aws4_request
d9c53b745da19f10565cf77b6b11fde944354448322b85d99b7f502a363fb57e
Signature:
e29af9cf031ac2c94a109f20fa9f8ad5b88b89686f86e32015c8b535b84fcc46
Header:
x-amz-date = Tue, 09 Oct 2018 05:55:08 GMT
Authorization = AWS4-HMAC-SHA256
Credential=xxxxxxxxxxxxxx/20181009/us-east-2/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=e29af9cf031ac2c94a109f20fa9f8ad5b88b89686f86e32015c8b535b84fcc46
================================

Issues while configuring the API Subscription BPS WSO2

So, i've my WSO2 BPS 3.6.0 configured to support SSL and a custom hostname i.e. mydomain.domain.com:9445 etc. and i'm trying to implement the API Subscription Workflow by following this documentation.
Now i've performed the following steps:
set the offset of wso2 bps to 2 and it is running fine with port: 9445
edited the wsa:Address tag in bothSubscriptionService.epr and SubscriptionCallbackService.epr located in API-M_HOME/business-processes/epr
as the bps server had a customized hostname instead of localhost (not sure if performing this step was right)
SubscriptionService.epr
SubscriptionCallBackService.epr
copy-pasted the epr folder from API-M_HOME/business-processes/epr to BPS_HOME/repository/conf/epr
Added the required BPEL package and human task accordingly
Navigated to the carbon console from APIM and edited the workflow-extensions.xml, here's how it looks like
set the TaskCoordinationEnabled tag of b4p-cordination-config.xml to true located in BPS_Home\repository\conf
Consider OTHER required configurations:
At API Manager End:
site.json file located at APIM_Home\repository\deployment\server\jaggeryapps\admin\site\conf
{
"theme": {
"base": "wso2",
"subtheme": "modern"
},
"context": "/admin",
"request_url": "READ_FROM_REQUEST",
"tasksPerPage": 10,
"allowedPermission": "/permission/admin/manage/apim_admin",
"workflows": {
"workFlowServerURL": "https://mydomain.domain.com:9445/services/",
},
"ssoConfiguration": {
"enabled": "false",
"issuer": "API_WORKFLOW_ADMIN",
"identityProviderURL": "https://localhost:9443/samlsso",
"keyStorePassword": "",
"identityAlias": "",
"keyStoreName": "",
"verifyAssertionValidityPeriod": "true",
"audienceRestrictionsEnabled": "true",
"responseSigningEnabled": "true",
"assertionSigningEnabled": "true",
"assertionEncryptionEnabled": "false",
"idpInit" : "false",
"idpInitSSOURL" : "https://localhost:9443/samlsso?spEntityID=API_WORKFLOW_ADMIN",
"externalLogoutPage" : "https://localhost:9443/samlsso?slo=true"
},
"reverseProxy": {
"enabled": false,
// values true , false , "auto" - will look for X-Forwarded-* headers
"host": "sample.proxydomain.com",
// If reverse proxy do not have a domain name use IP
"context": ""
//"regContext":"" // Use only if different path is used for registry
}
}
the workflowconfiguration in api-manager.xml
At BPS end:
carbon.xml
Issue: Now whenever a user navigates to APIM Store and subscribes to any API, the subscription request is listed at the APIM Admin console. When i select APPROVE from the provided ddl and click on the COMPLETE button, the record vanishes. However, this is the error that i see at WSO2's CMD windows:
APIM's cmd window
[2017-11-09 00:13:17,022] INFO - TimeoutHandler This engine will
expire all cal lbacks after GLOBAL_TIMEOUT: 120 seconds, irrespective
of the timeout action, af ter the specified or optional timeout
[2017-11-09 00:13:17,164] ERROR - TargetHandler I/O error: Host name
verificatio n failed for host : localhost javax.net.ssl.SSLException:
Host name verification failed for host : localhost
at org.apache.synapse.transport.http.conn.ClientSSLSetupHandler.verify(C
lientSSLSetupHandler.java:171)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession
.java:308)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSes
sion.java:410)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(Abstra
ctIODispatch.java:119)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor
.java:159)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(Abstr
actIOReactor.java:338)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(Abst
ractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIO
Reactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.
java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.
run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Thread.java:745)
[2017-11-09 00:13:17,188] WARN - EndpointContext Endpoint : AnonymousEndpoint w
ith address
https://localhost:9443/store/site/blocks/workflow/workflow-listener/
ajax/workflow-listener.jag will be marked SUSPENDED as it failed
[2017-11-09 00:13:17,193] WARN - EndpointContext Suspending endpoint
: Anonymou sEndpoint with address
https://localhost:9443/store/site/blocks/workflow/workflo
w-listener/ajax/workflow-listener.jag - current suspend duration is :
30000ms - Next retry after : Thu Nov 09 00:13:47 EST 2017
[2017-11-0900:13:17,201] INFO - LogMediator STATUS = Executing default 'fault'
sequence, ERROR_CODE = 101500, ERROR_MESSAGE = Error in Sender
[2017-11-09 00:14:17,238] INFO - SourceHandler Writer null when
calling informW riterError [2017-11-09 00:14:17,238] WARN -
SourceHandler Connection time out after reques t is read:
http-incoming-1 Socket Timeout : 60000 Remote Address : /10.10.30.130
:49249
[2017-11-09 00:14:24,671] ERROR - AxisEngine The endpoint
reference (EPR) for th e Operation not found is
/services/WorkflowCallbackService and the WSA Action = null. If this
EPR was previously reachable, please contact the server administra
tor. org.apache.axis2.AxisFault: The endpoint reference (EPR) for the
Operation not f ound is /services/WorkflowCallbackService and the WSA
Action = null. If this EPR was previously reachable, please contact
the server administrator.
at org.apache.axis2.engine.DispatchPhase.checkPostConditions(DispatchPha
se.java:102)
at org.apache.axis2.engine.Phase.invoke(Phase.java:329)
at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEn
closingRESTHandler(ServerWorker.java:325)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.j
ava:158)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(Native
WorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617)
at java.lang.Thread.run(Thread.java:745) [2017-11-09 00:14:24,673] ERROR - ServerWorker Error processing GET request for :
/services/WorkflowCallbackService org.apache.axis2.AxisFault: The
endpoint reference (EPR) for the Operation not f ound is
/services/WorkflowCallbackService and the WSA Action = null. If this
EPR was previously reachable, please contact the server
administrator.
at org.apache.axis2.engine.DispatchPhase.checkPostConditions(DispatchPha
se.java:102)
at org.apache.axis2.engine.Phase.invoke(Phase.java:329)
at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEn
closingRESTHandler(ServerWorker.java:325)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.j
ava:158)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(Native
WorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617)
at java.lang.Thread.run(Thread.java:745)
BPS's cmd window:
[2017-11-09 00:14:16,738] ERROR {org.wso2.carbon.bpel.core.ode.integration.Partn erService} - Error
sending message to Axis2 for ODE mex {PartnerRoleMex#hqejbhc
nphrcr2a32g83oh [PID
{http://workflow.subscription.apimgt.carbon.wso2.org}Subscr
iptionApprovalWorkFlowProcess-1] calling
org.apache.ode.bpel.epr.WSAEndpoint#705 fc38f.resumeEvent(...) Status
REQUEST} org.apache.axis2.AxisFault: Read timed out
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.jav
a:199)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessa
geWithCommons(CommonsHTTPTransportSender.java:451)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(Com
monsHTTPTransportSender.java:278)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(O
utOnlyAxisOperation.java:297)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:
149)
at org.wso2.carbon.bpel.core.ode.integration.utils.AxisServiceUtils.invo
keService(AxisServiceUtils.java:323)
at org.wso2.carbon.bpel.core.ode.integration.PartnerService.invoke(Partn
erService.java:333)
at org.wso2.carbon.bpel.core.ode.integration.BPELMessageExchangeContextI
mpl.invokePartner(BPELMessageExchangeContextImpl.java:43)
at org.apache.ode.bpel.engine.BpelRuntimeContextImpl.invoke(BpelRuntimeC
ontextImpl.java:897)
at org.apache.ode.bpel.runtime.INVOKE.run(INVOKE.java:130)
at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:4
51)
at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java:139)
at org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(BpelRuntime
ContextImpl.java:1002)
at org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeInstance(Partn
erLinkMyRoleImpl.java:250)
at org.apache.ode.bpel.engine.BpelProcess$1.invoke(BpelProcess.java:288)
at org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java
:224)
at org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java
:279)
at org.apache.ode.bpel.engine.BpelProcess.handleJobDetails(BpelProcess.j
ava:434)
at org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineIm
pl.java:558)
at org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerIm
pl.java:467)
at org.apache.ode.scheduler.simple.SimpleScheduler$RunJob$1.call(SimpleS
cheduler.java:633)
at org.apache.ode.scheduler.simple.SimpleScheduler$RunJob$1.call(SimpleS
cheduler.java:627)
at org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(Simpl
eScheduler.java:298)
at org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(Simpl
eScheduler.java:253)
at org.apache.ode.scheduler.simple.SimpleScheduler$RunJob.call(SimpleSch
eduler.java:627)
at org.apache.ode.scheduler.simple.SimpleScheduler$RunJob.call(SimpleSch
eduler.java:611)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:961)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:918)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:
78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106
)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.
java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$Http
ConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMetho
dBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodB
ase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.j
ava:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Htt
pMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMe
thodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav
a:397)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(Abst
ractHTTPSender.java:659)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.jav
a:195)
... 34 more
What could be the issue here? Any idea? do let me know. Thanks
Note that the bps workflow for API STATE CHANGE works just fine with the same configurations
Please note, that you are using calls with HTTPS with specific domain names
Host name verification failed for host : localhost at org.apache.synapse.transport.http.conn.ClientSSLSetupHandler.verify(ClientSSLSetupHandler.java:171) at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession .java:308)
the certificate provided is CN=localhost, so indeed the host verification fails
what you can do about it
simplest way is switching to http when on secure network (behind firewall, vpn, ..)
update SSL certificates of BPS and APIM to match their hostnames and they have to trust each others certificate (or certificate issuer)
disable SSL hostname validation in axis2.xml (I do not recommend it, good for DEV, VERY BAD for PROD) - set <parameter name="HostnameVerifier">AllowAll</parameter>