I have set up Netflix Eureka, Hystrix and Turbine on Cloud Foundry split in two apps:
A monitoring app "mrc-service" includes Eureka Server, Turbine and Hystrix Dashboard. The application.yml for this app looks like this:
---
spring:
profiles: cloud
eureka:
instance:
nonSecurePort: 80
hostname: ${vcap.application.uris[0]}
leaseRenewalIntervalInSeconds: 10
metadataMap:
instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}
client:
registerWithEureka: true
fetchRegistry: true
service-url:
defaultZone: https://mrc-service.myurl/eureka/
turbine:
aggregator:
clusterConfig: LOG-TEST
appConfig: log-test
The Hystrix stream producing app called "log-test" has multiple instances on Cloud Foundry. The app is an Eureka Client and exposes a Hystrix Stream using Spring Actuator. Here the application.yml for the app:
---
spring:
profiles: cloud
eureka:
instance:
nonSecurePort: 80
hostname: ${vcap.application.uris[0]}
metadataMap:
instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}
secure-port-enabled: true
client:
healthcheck:
enabled: true
service-url:
defaultZone: https://mrc-service.myurl/eureka/
The two instances of the log-test app register correctly with the Eureka server:
But when I start to monitor the turbine stream the Hystrix dashboard shows only one host (as indicated by the red arrow) instead of two:
The Turbine log retrieves both instances correctly, but then says that only one Host is up:
2017-08-23T10:12:10.764+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery : Fetching instances for app: log-test
2017-08-23T10:12:10.764+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery : Received instance list for app: log-test, size=2
2017-08-23T10:12:10.764+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.763 INFO 19 --- [ Timer-0] o.s.c.n.t.CommonsInstanceDiscovery : Fetching instance list for apps: [log-test]
2017-08-23T10:12:10.764+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Retrieved hosts from InstanceDiscovery: 2
2017-08-23T10:12:10.765+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Found hosts that have been previously terminated: 0
2017-08-23T10:12:10.765+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 DEBUG 19 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Retrieved hosts from InstanceDiscovery: [StatsInstance [hostname=log-test.myurl:80, cluster: LOG-TEST, isUp: true, attrs={securePort=443, fusedHostPort=log-test.myurl:443, instanceId=log-test:97d83c44-8b9e-44c4-56b4-742cef7bada0, port=80}], StatsInstance [hostname=log-test.myurl:80, cluster: LOG-TEST, isUp: true, attrs={securePort=443, fusedHostPort=log-test.myurl:443, instanceId=log-test:3d8359e4-a5c1-4aa0-5109-5b49a77a1f6f, port=80}]]
2017-08-23T10:12:10.765+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Hosts up:1, hosts down: 0
So I wonder if Turbine actually aggregates the Hystrix streams of the two instances. Turbine would have to contact the instances e.g. using Cloud Foundry specific header parameters like X-CF-APP-INSTANCE. Not sure if this already this happens.
Is the described approach even feasible on Cloud Foundry or do I have to use Turbine Stream with RabbitMQ instead?
I got an official reply from the Spring Cloud Netflix Issue tracker: aggregation of Hystrix data from multiple app instances on Cloud Foundry requires Turbine Stream in combination with a broker (e.g. RabbitMQ).
To open Turbine in aggregate way, it's the same steps as Hystrix, but you should inform the cluster via Turbine: http://localhost:8989//turbine.stream?cluster=READ.
That will open the same screen that Hystrix, but if I have more services, they will appear in an aggregate way.
Related
I have deployed a AWS managed Hyperledger Fabric v1.4.7 blockchain. The HLF blockchain network and the EC2 instance (hlf-client) are in the same VPC and everything seems to be working fine since I am able to invoke transactions using the cli container.
I have my client-app which is using fabric-sdk-go gateway API to connect to the fabric network using the connection-profile.yamlto invoke/query the blockchain. This client-app is running in a docker container on same EC2 instance as the cli container which has all the necessary security configuration. The client-app is unable to connect to the fabric network due to a bad certificate error
The error log on the client app is:
[fabsdk/util] 2021/11/02 09:55:17 UTC - lazyref.(*Reference).refreshValue -> WARN Error - initializer returned error: QueryBlockConfig failed: QueryBlockConfig failed: queryChaincode failed: Transaction processing for endorser [nd-cjfwwnimujabllevl6yitqqmxi.m-l3ascxxbincwrbtirbgpp4bp7u.n-rh3k6kahfnd6bgtxxgru7c3b5q.managedblockchain.ap-southeast-1.amazonaws.com:30003]: Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [nd-cjfwwnimujabllevl6yitqqmxi.m-l3ascxxbincwrbtirbgpp4bp7u.n-rh3k6kahfnd6bgtxxgru7c3b5q.managedblockchain.ap-southeast-1.amazonaws.com:30003]: connection is in TRANSIENT_FAILURE. Will retry again later
The corresponding peer log is:
[36m2021-11-02 10:07:17.789 UTC [grpc] handleRawConn -> DEBU 39501a[0m grpc: Server.Serve failed to complete security handshake from "10.0.2.131:39100": remote error: tls: bad certificate
[31m2021-11-02 10:10:17.809 UTC [core.comm] ServerHandshake -> ERRO 395322[0m TLS handshake failed with error remote error: tls: bad certificate server=PeerServer remoteaddress=10.0.2.131:12696
While invoking transactions using the cli the same certificate files are used. Could anyone tell me what's wrong with my setup here or am I missing any other configuration?
I have generated the ccp (connection-profile.yaml) as below:
---
name: n-RH3K6KAHFND6BGTXXGRU7C3B5Q
version: 1.0.0
client:
organization: Org1
connection:
timeout:
peer:
endorser: "300"
channels:
mychannel:
peers:
nd-CJFWWNIMUJABLLEVL6YITQQMXI:
endorsingPeer: true
chaincodeQuery: true
ledgerQuery: true
eventSource: true
organizations:
Org1:
mspid: m-L3ASCXXBINCWRBTIRBGPP4BP7U
peers:
- nd-CJFWWNIMUJABLLEVL6YITQQMXI
certificateAuthorities:
- m-L3ASCXXBINCWRBTIRBGPP4BP7U
peers:
nd-CJFWWNIMUJABLLEVL6YITQQMXI:
url: grpcs://nd-cjfwwnimujabllevl6yitqqmxi.m-l3ascxxbincwrbtirbgpp4bp7u.n-rh3k6kahfnd6bgtxxgru7c3b5q.managedblockchain.managedblockchain.us-east-1.amazonaws.com:30003
eventUrl: grpcs://nd-cjfwwnimujabllevl6yitqqmxi.m-l3ascxxbincwrbtirbgpp4bp7u.n-rh3k6kahfnd6bgtxxgru7c3b5q.managedblockchain.managedblockchain.us-east-1.amazonaws.com:30004
grpcOptions:
ssl-target-name-override: nd-CJFWWNIMUJABLLEVL6YITQQMXI
tlsCACerts:
path: /home/ec2-user/managedblockchain-tls-chain.pem
certificateAuthorities:
m-L3ASCXXBINCWRBTIRBGPP4BP7U:
url: https://ca.m-l3ascxxbincwrbtirbgpp4bp7u.n-rh3k6kahfnd6bgtxxgru7c3b5q.managedblockchain.managedblockchain.us-east-1.amazonaws.com:30002
httpOptions:
verify: false
tlsCACerts:
path: /home/ec2-user/managedblockchain-tls-chain.pem
caName: m-L3ASCXXBINCWRBTIRBGPP4BP7U
The following solution applies to:
HLF v1.4.7 AWS Managed Blockchain
Fabric client [fabric-sdk-go v1.0.0] Gateway programming model
To resolve the issue just remove the grpcOptions stanza
I have connected to an AWS instance which was set up for MarkLogic using the AWS Systems Manager. I am trying to start the MarkLogic Server, but I am receiving the following error response:
Set configuration: JAVA_HOME="/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.252.b09-2.amzn2.0.1.x86_64"
Set configuration: MARKLOGIC_MDB_TYPE=""
Set configuration: AWS_REGION="ap-southeast-2"
Set configuration: AWS_DEFAULT_REGION="ap-southeast-2"
Set configuration: MARKLOGIC_ZONE="ap-southeast-2a"
Initialize Configuration.
AWS Region: ap-southeast-2, ZONE: ap-southeast-2a. INSTANCE: i-08c0992c858711a67
Instance is not managed
Waiting for device mounted to come online : /dev/nvme1n1
Volume /dev/sdf has failed to attach - aborting
Warning: ec2-startup did not complete successfully
Check the error logs for details
Starting MarkLogic: [FAILED]
This was the output on the log for mlcmd:
"2020-08-17 02:10:26,821 0 INFO [main] shell.Shell - xmlsh initialize
"2020-08-17 02:10:26,952 131 INFO [main] builtin.log - loading init.xsh
"2020-08-17 02:10:27,102 281 INFO [main] builtin.log - initializing mlcmd
"2020-08-17 02:10:27,103 282 INFO [main] builtin.log - loading /var/local/mlcmd.conf
"2020-08-17 02:10:27,297 476 TRACE [main] mlcmd.trace - init-config: exit-status: 1 args: Not loading mdb functions - not a managed cluster
"2020-08-17 02:10:27,299 478 TRACE [main] mlcmd.trace - complete init.xsh: exit-status: 1 args:
"2020-08-17 02:10:27,299 478 INFO [main] builtin.log - runing init-config.xsh
"2020-08-17 02:10:27,942 0 INFO [main] shell.Shell - xmlsh initialize
"2020-08-17 02:10:28,042 100 INFO [main] builtin.log - loading init.xsh
"2020-08-17 02:10:28,173 231 INFO [main] builtin.log - initializing mlcmd
"2020-08-17 02:10:28,174 232 INFO [main] builtin.log - loading /var/local/mlcmd.conf
"2020-08-17 02:10:28,387 445 TRACE [main] mlcmd.trace - ec2-startup: exit-status: 1 args: Not loading mdb functions - not a managed cluster
"2020-08-17 02:10:28,389 447 TRACE [main] mlcmd.trace - complete init.xsh: exit-status: 1 args:
How do I resolve this issue?
If there is more required information, do let me know and I will try to get it
It appears that you are attempting to start a self-managed instance/cluster, while the Managed Cluster feature has not been disabled.
The MarkLogic Managed Cluster feature is the recommended way to deploy a MarkLogic Cluster on AWS so it is enabled by default. Managed clusters are meant to be deployed using the MarkLogic CloudFormation Templates.
Deploying MarkLogic on EC2 Using CloudFormation
The Managed Cluster feature reduces the amount of work necessary to setup the initial cluster, and creates an Auto Scaling Group that will automatically re-launch an instance that gets terminated, and when MarkLogic starts on the new instance, it will remount the associated EBS data drive.
CloudFormation Template Overview
If you wish to have a self-managed cluster, then you will need to create an /etc/marklogic.conf file to disable the feature at startup.
AWS Configuration Variables
Best Practice Editing MarkLogic Server Environment Variables
I would recommend reviewing the following guide, as it details using both the Managed Cluster feature, as well as self-managed clusters.
MarkLogic Server on Amazon Web Services (AWS) Guide
Trying to use aws cli to push a docker container. Getting connection reset. Not sure why?
ecs-cli compose --project-name blah service up --create-log-groups --cluster-config blah --timeout 20
WARN[0000] Skipping unsupported YAML option for service... option name=expose service name=app
WARN[0000] Ignoring the ip address while transforming it to task definition container=app portMapping="0.0.0.0:8080:8080"
WARN[0000] Ignoring the ip address while transforming it to task definition container=app portMapping="0.0.0.0:8080:8080"
INFO[0000] Using ECS task definition TaskDefinition="blah:2"
WARN[0000] No log groups to create; no containers use 'awslogs'
INFO[0001] Updated ECS service successfully desiredCount=1 force-deployment=false service=blah
INFO[0047] (service blah) has started 1 tasks: (task d4d52496-057a-4b24-878a-4cf654085eff). timestamp="2019-05-24 18:15:02 +0000 UTC"
INFO[0125] (service blah) has started 1 tasks: (task 9484000f-4a4d-4b2d-8fdb-a9668012d6ae). timestamp="2019-05-24 18:16:23 +0000 UTC"
INFO[0202] (service blah) has started 1 tasks: (task f13dea68-e996-40ce-a44f-be266219b001). timestamp="2019-05-24 18:17:32 +0000 UTC"
ERRO[0392] Error describing service error="RequestError: send request failed\ncaused by: Post https://ecs.us-west-2.amazonaws.com/: read tcp 192.168.1.231:53094->52.119.169.134:443: read: connection reset by peer" service=blah
FATA[0392] RequestError: send request failed
caused by: Post https://ecs.us-west-2.amazonaws.com/: read tcp 192.168.1.231:53094->52.119.169.134:443: read: connection reset by peer
I am deploying SpringBoot application in PCF which has oracle database connection i have also made user created service instance with oracle credentials and binded to application.
Following is the VCAP service variables :
{
"name": "healthwatch-api-database",
"instance_name": "healthwatch-api-database",
"binding_name": null,
"credentials": {
"driver": "oracle.jdbc.OracleDriver",
"url": "jdbc:oracle:thin:#(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=10.157.129.175)(PORT=1527))(CONNECT_DATA=(SERVER=DEDICATED)(SID=DEVCLOUD)))",
"username": "EXTRANET_USER",
"password": "EXTRANET_USER1"
},
Following are the logs after pushing application :
2018-10-09T18:26:41.29+0530 [APP/PROC/WEB/0] OUT o.s.b.w.s.ServletRegistrationBean - Mapping servlet: 'dispatcherServlet' to [/]
2018-10-09T18:26:41.47+0530 [APP/PROC/WEB/0] OUT 09 Oct 2018 12:56:41.471/UTC [main] INFO
2018-10-09T18:26:41.47+0530 [APP/PROC/WEB/0] OUT o.s.j.d.DriverManagerDataSource - Loaded JDBC driver: oracle.jdbc.OracleDriver
2018-10-09T18:26:42.17+0530 [APP/PROC/WEB/0] OUT 09 Oct 2018 12:56:42.174/UTC [main] WARN
2018-10-09T18:26:42.17+0530 [APP/PROC/WEB/0] OUT o.s.b.a.orm.jpa.DatabaseLookup - Unable to determine jdbc url from datasource
2018-10-09T18:26:42.17+0530 [APP/PROC/WEB/0] OUT org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1
2018-10-09T18:26:42.17+0530 [APP/PROC/WEB/0] OUT ORA-01882: timezone region not found
2018-10-09T18:26:42.17+0530 [APP/PROC/WEB/0] OUT at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:338)
Looking at the error log, the cause seems to be related to timezone settings.
2018-10-09T18:26:42.17+0530 [APP/PROC/WEB/0] OUT ORA-01882: timezone region not found
Here is a post talking about a similar issue - Getting ORA-01882: timezone region not found with Oracle UCP, on aws ec2 instance?
Now how do you pass the timezone to your app?
You do that through manifest file. Check out the article -
https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html
When I try to deploy SpringMvcGroovy-0.1.0.jar to CloudFoundry with 'cf push SpringMvcGroovy -p libs/SpringMvcGroovy-0.1.0.jar' command, it uploads droplet successfully but fails to start the instance:
...
2014-06-11T03:04:43.37-0400 [App/0] OUT 2014-06-11 07:04:43.373 INFO 33 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port: 8080
2014-06-11T03:04:43.37-0400 [App/0] OUT 2014-06-11 07:04:43.376 INFO 33 --- [ main] c.b.springmvc.GreetingApplication : Started GreetingApplication in 5.263 seconds (JVM running for 6.868)
2014-06-11T03:05:36.53-0400 [DEA] OUT Instance (index 0) failed to start accepting connections
2014-06-11T03:05:36.54-0400 [API] OUT App instance exited with guid 83d01863-9813-4b7b-8113-a2a8fcc8a43d payload: {"cc_partition"=>"default", "droplet"=>"83d01863-9813-4b7b-8113-a2a8fcc8a43d", "version"=>"9baf1eef-275f-4000-bcde-f0810985e9a6", "instance"=>"31ec2e2f0a704d2cb7555bfbbcb26aa0", "index"=>0, "reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to accept connections within health check timeout", "crash_timestamp"=>1402470336}
2014-06-11T03:05:36.65-0400 [App/0] ERR
What could be the reason for that?
The app seems to be starting on port 8080 (which obviously isn't going to work). The buildpack ought to detect a Spring Boot app and set the server.port argument, so something is broken there (raise an issue with them if you want to get it fixed: https://github.com/cloudfoundry/java-buildpack/issues?state=open). You can work around it be setting SERVER_PORT=${PORT} yourself as an environment variable.
EDIT: I wouldn't bother raising any issues. The app works for me if I upgrade to Spring Boot 1.1.0.RELEASE.