Unable to start Marklogic service on AWS - amazon-web-services

I have connected to an AWS instance which was set up for MarkLogic using the AWS Systems Manager. I am trying to start the MarkLogic Server, but I am receiving the following error response:
Set configuration: JAVA_HOME="/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.252.b09-2.amzn2.0.1.x86_64"
Set configuration: MARKLOGIC_MDB_TYPE=""
Set configuration: AWS_REGION="ap-southeast-2"
Set configuration: AWS_DEFAULT_REGION="ap-southeast-2"
Set configuration: MARKLOGIC_ZONE="ap-southeast-2a"
Initialize Configuration.
AWS Region: ap-southeast-2, ZONE: ap-southeast-2a. INSTANCE: i-08c0992c858711a67
Instance is not managed
Waiting for device mounted to come online : /dev/nvme1n1
Volume /dev/sdf has failed to attach - aborting
Warning: ec2-startup did not complete successfully
Check the error logs for details
Starting MarkLogic: [FAILED]
This was the output on the log for mlcmd:
"2020-08-17 02:10:26,821 0 INFO [main] shell.Shell - xmlsh initialize
"2020-08-17 02:10:26,952 131 INFO [main] builtin.log - loading init.xsh
"2020-08-17 02:10:27,102 281 INFO [main] builtin.log - initializing mlcmd
"2020-08-17 02:10:27,103 282 INFO [main] builtin.log - loading /var/local/mlcmd.conf
"2020-08-17 02:10:27,297 476 TRACE [main] mlcmd.trace - init-config: exit-status: 1 args: Not loading mdb functions - not a managed cluster
"2020-08-17 02:10:27,299 478 TRACE [main] mlcmd.trace - complete init.xsh: exit-status: 1 args:
"2020-08-17 02:10:27,299 478 INFO [main] builtin.log - runing init-config.xsh
"2020-08-17 02:10:27,942 0 INFO [main] shell.Shell - xmlsh initialize
"2020-08-17 02:10:28,042 100 INFO [main] builtin.log - loading init.xsh
"2020-08-17 02:10:28,173 231 INFO [main] builtin.log - initializing mlcmd
"2020-08-17 02:10:28,174 232 INFO [main] builtin.log - loading /var/local/mlcmd.conf
"2020-08-17 02:10:28,387 445 TRACE [main] mlcmd.trace - ec2-startup: exit-status: 1 args: Not loading mdb functions - not a managed cluster
"2020-08-17 02:10:28,389 447 TRACE [main] mlcmd.trace - complete init.xsh: exit-status: 1 args:
How do I resolve this issue?
If there is more required information, do let me know and I will try to get it

It appears that you are attempting to start a self-managed instance/cluster, while the Managed Cluster feature has not been disabled.
The MarkLogic Managed Cluster feature is the recommended way to deploy a MarkLogic Cluster on AWS so it is enabled by default. Managed clusters are meant to be deployed using the MarkLogic CloudFormation Templates.
Deploying MarkLogic on EC2 Using CloudFormation
The Managed Cluster feature reduces the amount of work necessary to setup the initial cluster, and creates an Auto Scaling Group that will automatically re-launch an instance that gets terminated, and when MarkLogic starts on the new instance, it will remount the associated EBS data drive.
CloudFormation Template Overview
If you wish to have a self-managed cluster, then you will need to create an /etc/marklogic.conf file to disable the feature at startup.
AWS Configuration Variables
Best Practice Editing MarkLogic Server Environment Variables
I would recommend reviewing the following guide, as it details using both the Managed Cluster feature, as well as self-managed clusters.
MarkLogic Server on Amazon Web Services (AWS) Guide

Related

JanusGraph: access Amazon Managed Cassandra from EC2

I'm trying to set up JanusGraph to access Amazon MCS. The infrastructure is all there to allow access, but I'm facing difficulties at the config step.
This is the config I'm setting for janusgraph-cql.properties:
storage.backend=cql
storage.hostname=cassandra.ap-southeast-1.amazonaws.com
storage.port=9142
storage.username=${CASSANDRA_USERNAME}
storage.password=${CASSANDRA_PASSWORD}
storage.cql.ssl.truststore.location=${CASSANDRA_TRUSTSTORE_LOCATION}
storage.cql.ssl.truststore.password=${CASSANDRA_TRUSTSTORE_PASSWORD}
storage.cql.ssl.enabled=true
Amazon MCS exposes port 9142, instead of 9402.
When I start gremlin-server.sh, I can see the following outputs:
2897 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /13.251.116.141:9042 added
2898 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /13.251.117.140:9042 added
2898 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /13.251.117.134:9042 added
2898 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /13.251.116.137:9042 added
2898 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /13.251.116.182:9042 added
2899 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host cassandra.ap-southeast-1.amazonaws.com/13.251.117.0:9142 added
2899 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /13.251.116.84:9042 added
2899 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /13.251.117.219:9042 added
2899 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /13.251.116.144:9042 added
2899 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /13.251.116.1:9042 added
Even though I've set the port to 9142, new Cassandra hosts with port 9042 are still being added, making the process fail (since 9042 is not available). Is there something I'm doing wrong?

CockroachDB on AWS EKS cluster - [n?] no stores bootstrapped

I am attempting to deploy CockroachDB:v2.1.6 to a new AWS EKS cluster. Everything is deployed successfully; statefulset, services, pv's & pvc's are created. The AWS EBS volumes are created successfully too.
The issue is the pods never get to a READY state.
pod/cockroachdb-0 0/1 Running 0 14m
pod/cockroachdb-1 0/1 Running 0 14m
pod/cockroachdb-2 0/1 Running 0 14m
If I 'describe' the pods I get the following:
Normal Pulled 46s kubelet, ip-10-5-109-70.eu-central-1.compute.internal Container image "cockroachdb/cockroach:v2.1.6" already present on machine
Normal Created 46s kubelet, ip-10-5-109-70.eu-central-1.compute.internal Created container cockroachdb
Normal Started 46s kubelet, ip-10-5-109-70.eu-central-1.compute.internal Started container cockroachdb
Warning Unhealthy 1s (x8 over 36s) kubelet, ip-10-5-109-70.eu-central-1.compute.internal Readiness probe failed: HTTP probe failed with statuscode: 503
If I examine the logs of a pod I see this:
I200409 11:45:18.073666 14 server/server.go:1403 [n?] no stores bootstrapped and --join flag specified, awaiting init command.
W200409 11:45:18.076826 87 vendor/google.golang.org/grpc/clientconn.go:1293 grpc: addrConn.createTransport failed to connect to {cockroachdb-0.cockroachdb:26257 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup cockroachdb-0.cockroachdb on 172.20.0.10:53: no such host". Reconnecting...
W200409 11:45:18.076942 21 gossip/client.go:123 [n?] failed to start gossip client to cockroachdb-0.cockroachdb:26257: initial connection heartbeat failed: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup cockroachdb-0.cockroachdb on 172.20.0.10:53: no such host"
I came across this comment from the CockroachDB forum (https://forum.cockroachlabs.com/t/http-probe-failed-with-statuscode-503/2043/6)
Both the cockroach_out.log and cockroach_output1.log files you sent me (corresponding to mycockroach-cockroachdb-0 and mycockroach-cockroachdb-2) print out no stores bootstrapped during startup and prefix all their log lines with n?, indicating that they haven’t been allocated a node ID. I’d say that they may have never been properly initialized as part of the cluster.
I have deleted everything including pv's, pvc's & AWS EBS volumes through the kubectl delete command and reapplied with the same issue.
Any thoughts would be very much appreciated. Thank you
I was not aware that you had to initialize the CockroachDB cluster after creating it. I did the following to resolve my issue:
kubectl exec -it cockroachdb-0 -n /bin/sh
/cockroach/cockroach init
See here for more details - https://www.cockroachlabs.com/docs/v19.2/cockroach-init.html
After this the pods started running correctly.

Error when running cassandra on Google Cloud on external ip - Failed to bind port 9042 on 34.89.109.98

I am trying to make Cassandra run on Google Cloud using external ip of the VM. But I am getting error Failed to bind port 9042 on 34.89.109.98. As far as I can see, I have followed the rules of setting firewall rules but I am still not able to resolve the issue. I have attached the pics of my configuration for your reference.
1) The firewall rule is
2) The list of all the rules is
3) The VM is
More Information
I followed the steps in https://linuxize.com/post/how-to-install-apache-cassandra-on-debian-9/ to install Cassandra. This automatically started cassandra. Then I killed cassandra, changed the ip address to external IP in cassandra.yaml file and started it again. It didn't work. Then I started working around with VPN settings.
Part of the message dump after I issue the command to start cassandra /usr/sbin/cassandra -f
INFO [main] 2019-12-18 16:09:40,755 StorageService.java:1521 - JOINING: Finish joining ring
INFO [main] 2019-12-18 16:09:40,826 StorageService.java:2442 - Node localhost/127.0.0.1 state jump to NORMAL
INFO [main] 2019-12-18 16:09:41,027 NativeTransportService.java:68 - Netty using native Epoll event loop
INFO [main] 2019-12-18 16:09:41,071 Server.java:158 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final
.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812
a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a
, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=
netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-
epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452
812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.4
4.Final.452812a]
INFO [main] 2019-12-18 16:09:41,071 Server.java:159 - Starting listening for CQL clients on /35.197.238.136:9042 (
unencrypted)...
Exception (java.lang.IllegalStateException) encountered during startup: Failed to bind port 9042 on 35.197.238.136.
java.lang.IllegalStateException: Failed to bind port 9042 on 35.197.238.136.
at org.apache.cassandra.transport.Server.start(Server.java:163)
at java.util.Collections$SingletonSet.forEach(Collections.java:4769)
at org.apache.cassandra.service.NativeTransportService.start(NativeTransportService.java:124)
at org.apache.cassandra.service.CassandraDaemon.startNativeTransport(CassandraDaemon.java:696)
at org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:546)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:635)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:742)
ERROR [main] 2019-12-18 16:09:41,100 CassandraDaemon.java:759 - Exception encountered during startup
java.lang.IllegalStateException: Failed to bind port 9042 on 35.197.238.136.
at org.apache.cassandra.transport.Server.start(Server.java:163) ~[apache-cassandra-3.11.5.jar:3.11.5]
at java.util.Collections$SingletonSet.forEach(Collections.java:4769) ~[na:1.8.0_232]
at org.apache.cassandra.service.NativeTransportService.start(NativeTransportService.java:124) ~[apache-cass
andra-3.11.5.jar:3.11.5]
at org.apache.cassandra.service.CassandraDaemon.startNativeTransport(CassandraDaemon.java:696) [apache-cass
andra-3.11.5.jar:3.11.5]
at org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:546) [apache-cassandra-3.11.5.ja
r:3.11.5]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:635) [apache-cassandra-3.11.5
.jar:3.11.5]
Within the Cassandra cassandra.yaml file you can bind your Cassandra server to an IP address on which it is listening. The default is 127.0.0.1 (localhost) and is not suitable for external connections.
The address values you can use are the addresses that the Compute Engine has associated with it. These can be discovered using:
ip addr
It is important to realize that a Compute Engine may appear to have a public IP address when shown in the GCP Console, but that is not a network interface on the Compute Engine. In the example in your original question, the Compute Engine IP address would be 10.154.0.4. This is the address you want to set in your configuration file.
See also this document which describes setting up Cassandra on GCP:
Spinning up a Cassandra Cluster on Google Cloud (for free) with just a browser

AWS BeanStalk doesn't tend to connect to AWS RDS

I think I am almost there.
I created an instance of AWS BeanStalk and added an oracle DB instance to it.
When I found the log, I saw the driver was loaded but it keeps saying that URL is
invalid.
Here are my RDS info and log message.
[RDS Info]
Endpoint = aa1c9autjaqoufk.c2k1ch01futy.ap-northeast-2.rds.amazonaws.com
Port = 1521
Public Access = yes
[System Log]
25-Jun-2018 02:42:56.759 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
25-Jun-2018 02:42:56.787 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
25-Jun-2018 02:42:56.796 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"]
25-Jun-2018 02:42:56.799 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
25-Jun-2018 02:42:56.800 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1366 ms
25-Jun-2018 02:42:56.842 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
25-Jun-2018 02:42:56.848 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.50
25-Jun-2018 02:42:56.872 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /var/lib/tomcat8/webapps/ROOT
25-Jun-2018 02:42:58.613 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
25-Jun-2018 02:42:58.689 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /var/lib/tomcat8/webapps/ROOT has finished in 1,817 ms
25-Jun-2018 02:42:58.693 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
25-Jun-2018 02:42:58.720 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
25-Jun-2018 02:42:58.736 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 1935 ms
Loading driver...
Driver loaded!
jdbc:oracle:oci://aa1c9autjaqoufk.c2k1ch01futy.ap-northeast-2.rds.amazonaws.com:1521/ebdb?user=username&password=password
SQLException: Invalid Oracle URL specified
SQLState: 99999
VendorError: 17067
Closing the connection.
SQLException: Invalid Oracle URL specified
SQLState: 99999
VendorError: 17067
Closing the connection.
I included ojdbc8 drvier in my web project library and made a build.
Is this about driver? What am I doing wrong?
Message clearly says your URL is incorrect,
It should be something like below.
//step1 load the driver class
Class.forName("oracle.jdbc.driver.OracleDriver");
//step2 create the connection object
Connection con=DriverManager.getConnection(
"jdbc:oracle:thin:#aa1c9autjaqoufk.c2k1ch01futy.ap-northeast-2.rds.amazonaws.com:1521:edb","username","password");
`

I can connect to AWS RDS via sqldeveloper but can't by Java application

It is so werid that I can connect to AWS RDS with sqldeveloper but can't with my java application(java source code or jsp)
When I try to access to RDS, there are errors like:
coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
26-Jun-2018 04:24:33.203 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
26-Jun-2018 04:24:33.212 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"]
26-Jun-2018 04:24:33.215 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
26-Jun-2018 04:24:33.219 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1387 ms
26-Jun-2018 04:24:33.265 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
26-Jun-2018 04:24:33.266 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.50
26-Jun-2018 04:24:33.286 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /var/lib/tomcat8/webapps/ROOT
26-Jun-2018 04:24:35.020 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
26-Jun-2018 04:24:35.097 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /var/lib/tomcat8/webapps/ROOT has finished in 1,811 ms
26-Jun-2018 04:24:35.100 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
26-Jun-2018 04:24:35.106 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
26-Jun-2018 04:24:35.108 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 1888 ms
Loading driver...
Driver loaded!
jdbc:oracle:thin://IP:1521/ORCL?user=username&password=password
SQLException: Invalid Oracle URL specified
SQLState: 99999
VendorError: 17067
Closing the connection.
SQLException: Invalid Oracle URL specified
SQLState: 99999
VendorError: 17067
Closing the connection.
But the URL is just the same value as I tried with sqldeveloper.
Is there anything wrong?
Please enlighten me since I've been suffering for this about a week! :(
I'm not sure how your application is set up, but I'm using Maven & Spring Boot and I got it working like this:
I mainly followed this guide, ignoring the .sql files, thymeleaf UI, "model.addAttribute("cities", cities);" part, and the html file:
https://zetcode.com/springboot/postgresql/
My application.properties file looks like this
postgres.comment.aa=https://zetcode.com/springboot/postgresql/
spring.main.banner-mode=off
logging.level.org.springframework=ERROR
spring.jpa.hibernate.ddl-auto=none
spring.datasource.initialization-mode=always
spring.datasource.platform=postgres
spring.datasource.url=jdbc:postgresql://your-rds-url-here.us-east-1.rds.amazonaws.com:yourDbPortHere/postgres
spring.datasource.username=postgres
spring.datasource.password=<your db password here>
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
If you have custom schemas, you can append "?currentSchema=users" to the url:
spring.datasource.url=jdbc:postgresql://your-rds-url-here.us-east-1.rds.amazonaws.com:yourDbPortHere/postgres?currentSchema=users
Thanks to this SO answer for the schema:
Is it possible to specify the schema when connecting to postgres with JDBC?
These other couple links also helped
https://turreta.com/2015/03/01/how-to-specify-a-default-schema-when-connecting-to-postgresql-using-jdbc/
https://doc.cuba-platform.com/manual-latest/db_schema_connection.html