I have launched an aws neptune instance and installed apache-tinkerpop-gremlin-console version 3.3.3 on windows 10 machine.
neptune-remote.yml looks like:
hosts: [abc-nept.XXXXXX.us-XXXX-1.neptune.amazonaws.com]
port: 8182
serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { serializeResultToString: true }}
after running gremlin.bat next command is:
:remote connect tinkerpop.server conf/neptune-remote.yaml
Now at this stage I am able to make queries and those are working! So question is how can I check whether I am actually connected to aws neptune instance or not?
I assume your question is related to having multiple :remote instances configured. Obviously, if you've simply created:
:remote connect tinkerpop.server conf/neptune-remote.yaml
then the only place your data could be going to or coming from is Neptune. The Console does allow multiple :remote instance that you can switch between so if you also had one for a local Gremlin Server then you might want to confirm which one you're sending requests. You just do this:
gremlin> :remote
==>Remote - Gremlin Server - [localhost/127.0.0.1:8182]
You'll be able to see the "current" :remote and thus know whether it is for Neptune or your local Gremlin Server instance.
Related
We have an application that is using AWS MemoryDB for Redis. We have setup a cluster with one shard and two nodes. One of the nodes (named 0001-001) is a primary read/write while the other one is a read replica (named 0001-002).
After deploying the application, connecting to MemoryDB sometimes fails when we use the cluster endpoint connection string to connect. If we restart the application a few times it suddenly starts working. It seems to be random when it succeeds or not. The error we get is the following:
Endpoint Unspecified/ourapp-memorydb-cluster-0001-001.ourapp-memorydb-cluster.xxxxx.memorydb.eu-west-1.amazonaws.com:6379 serving hashslot 6024 is not reachable at this point of time. Please check connectTimeout value. If it is low, try increasing it to give the ConnectionMultiplexer a chance to recover from the network disconnect. IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=0,Free=32767,Min=2,Max=32767), Local-CPU: n/a
If we connect directly to the primary read/write node we get no such errors.
If we connect directly to the read replica it always fails. It even gets the error above, compaining about the "0001-001" node.
We use .NET Core 6
We use Microsoft.Extensions.Caching.StackExchangeRedis 6.0.4 which depends on StackExchange.Redis 2.2.4
The application is hosted in AWS ECS
StackExchangeRedisCache is added to the service collection in a startup file :
services.AddStackExchangeRedisCache(o =>
{
o.InstanceName = redisConfiguration.Instance;
o.ConfigurationOptions = ToRedisConfigurationOptions(redisConfiguration);
});
...where ToRedisConfiguration returns a basic ConfigurationOptions object :
new ConfigurationOptions()
{
EndPoints =
{
{ "clustercfg.ourapp-memorydb-cluster.xxxxx.memorydb.eu-west-1.amazonaws.com", 6379 } // Cluster endpoint
},
User = "username",
Password = "password",
Ssl = true,
AbortOnConnectFail = false,
ConnectTimeout = 60000
};
We tried multiple shards with multiple nodes and it also sometimes fail to connect to the cluster. We even tried to update the dependency StackExchange.Redis to 2.5.43 but no luck.
We could "solve" it by directly connecting to the primary node, but if a failover occurs and 0001-002 becomes the primary node we would have to manually change our connection string, which is not acceptable in a production environment.
Any help or advice is appreciated, thanks!
As my last post at 403 Forbidden error for Gremlin to AWS Neptune, I could successfully connect to my Neptune Cluster DB via my Tinkerpop Gremlin console v 3.4.3 that installed at my EC2 instance as v 3.4.1 suggested at https://docs.aws.amazon.com/neptune/latest/userguide/iam-auth-connecting-gremlin-console.html didn't work for me.
\,,,/
(o o)
-----oOOo-(3)-oOOo-----
plugin activated: tinkerpop.server
plugin activated: tinkerpop.utilities
plugin activated: tinkerpop.tinkergraph
gremlin> :remote connect tinkerpop.server conf/neptune-remote.yaml
==>Configured <my neptune>.cluster-cm<cluster id>.ap-southeast-2.neptune.amazonaws.com/<private ip>:8182
gremlin> :remote console
==>All scripts will now be sent to Gremlin Server - [<my neptune>.cluster-cm<cluster id>.ap-southeast-2.neptune.amazonaws.com/<private ip>:8182] - type ':remote console' to return to local mode
However, I'm getting NoSuchMethodError error for all Gremlin commands (g.) that I used on the console.
e.g:
g.V()
gremlin> g.V()
org.apache.tinkerpop.gremlin.driver.RequestOptions$Builder.userAgent(Ljava/lang/String;)Lorg/apache/tinkerpop/gremlin/driver/RequestOptions$Builder;
Type ':help' or ':h' for help.
Display stack trace? [yN]Y
java.lang.NoSuchMethodError: org.apache.tinkerpop.gremlin.driver.RequestOptions$Builder.userAgent(Ljava/lang/String;)Lorg/apache/tinkerpop/gremlin/driver/RequestOptions$Builder;
at org.apache.tinkerpop.gremlin.console.jsr223.DriverRemoteAcceptor.send(DriverRemoteAcceptor.java:214)
at org.apache.tinkerpop.gremlin.console.jsr223.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:168)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:110)
...
g.addV('person').property('name', 'justin')
gremlin> g.addV('person').property('name', 'justin')
org.apache.tinkerpop.gremlin.driver.RequestOptions$Builder.userAgent(Ljava/lang/String;)Lorg/apache/tinkerpop/gremlin/driver/RequestOptions$Builder;
Type ':help' or ':h' for help.
Display stack trace? [yN]Y
java.lang.NoSuchMethodError: org.apache.tinkerpop.gremlin.driver.RequestOptions$Builder.userAgent(Ljava/lang/String;)Lorg/apache/tinkerpop/gremlin/driver/RequestOptions$Builder;
at org.apache.tinkerpop.gremlin.console.jsr223.DriverRemoteAcceptor.send(DriverRemoteAcceptor.java:214)
at org.apache.tinkerpop.gremlin.console.jsr223.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:168)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:110)
....
I have also tried the latest Apache Tinkerpop Gremlin Console 3.4.6, same error I had...
Thanks
I think the step you're missing is taking the temporary credentials provided by your EC2 instance's assigned IAM role and pushing those into the Default Credential Provider chain in order for them to be seen by the SigV4Channelizer used by the Gremlin Console. A high level overview of that process can be seen here: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html
A more prescriptive way of handling this for Neptune can be found here: https://docs.aws.amazon.com/neptune/latest/userguide/iam-auth-temporary-credentials.html See the section titled, "Setting Up Amazon EC2 for Neptune IAM Authentication".
I just tried to use Gremlin console 3.4.1 and it's working as expected... I think it's due to Incompatible Version issue. I was using Gremlin console 3.4.6
I am trying to connect to an AWS Redshift database from a lambda function using c#, dotnet core 2.0, and npgsql. I am having difficulty with SSL.
I have created two non-publicly-accessible Redshift databases in a dedicated VPC. The lambda executes in the same VPC. The two databases are identical in every way except that one has the "force SSL" parameter set to true.
Using the following code snippet, I can access the non-SSL database just fine:
using (var conn = new NpgsqlConnection ("Host=x; Port=5439; Username=x;
Password=x;Database=xxx")
{
Console.WriteLine("Redshift pre-Open!");
conn.Open();
Console.WriteLine("Redshift: post-Open!");
...
}
When I access the SSL database, I get the "missing hba.conf" error message - seems standard, I've seen it before ...
When I append to the connection string: "ssl Mode=Require;Server Compatibility Mode=Redshift;Trust Server Certificate=true"
the conn.open statement hangs, and the second write statement never shows up in cloudwatch.
And yet ... this connection statement works when accessing the same database thru a rest API and C#/dotnetcore 2 WEBAPI (same runtime environment), with
an EC2 instance and load balancer.
A Python lambda connecting to the SSL database, in the same environment - subnets, security groups, lambda triggers, lambda parameters, ... is working just fine.
The csproj references Amazon.Lambda.Core 1.0.0, Amazon.Lambda.Serialization.Json 1.1.0, and
Npgsql.EntityFrameworkCore.PostgreSQL 2.0.1.
I'd try Wireshark, maybe, in another environment - but running as a lambda, I'm not sure how best to debug. I've tried many permutations and combinations, and I wouldn't put it past myself to be missing something blindingly obvious,
but I absolutely do not see why hangs. Thank you.
I am trying to execute following flow:
user hits AWS Gateway (REST),
it triggers AWS Lambda,
that uses Tinkerpop/Gremlin connects to
TitanDB on EC2, that uses
AWS DynamoDB in cloud (not on EC2) as backend.
Right now I have managed to crete fully working TitanDB instance on EC2, that stores data in DynamoDB in cloud.
I am also able to connect from AWS Lambda to EC2 through Tinkerpop/Gremlin BUT only this way:
Cluster.build()
.addContactPoint("10.x.x.x") // ip of EC2
.create()
.connect()
.submit("here I type my query as string and it will work");
And this works, however I strongly prefer to use "Criteria API" (GremlinPipeline) instead of plain Gremlin language.
In other words, I need ORM or something like that.
I know, that Tinkerpop includes it.
I have realized, that what I need is object of class Graph.
This is what I have tried:
Graph graph = TitanFactory
.build()
.set("storage.hostname", "10.x.x.x")
.set("storage.backend", "com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager")
.set("storage.dynamodb.client.credentials.class-name", "com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
.set("storage.dynamodb.client.credentials.constructor-args", "")
.set("storage.dynamodb.client.endpoint", "https://dynamodb.ap-southeast-2.amazonaws.com")
.open();
However, it throws "Could not find implementation class: com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager".
Of course, computer is correct, as IntelliJ IDEA also cannot find it.
My dependencies:
//
// aws
compile 'com.amazonaws:aws-lambda-java-core:+'
compile 'com.amazonaws:aws-lambda-java-events:+'
compile 'com.amazonaws:aws-lambda-java-log4j:+'
compile 'com.amazonaws:aws-java-sdk-dynamodb:1.10.5.1'
compile 'com.amazonaws:aws-java-sdk-ec2:+'
//
// database
// titan 1.0.0 is compatible with gremlin 3.0.2-incubating, but not yet with 3.2.0
compile 'com.thinkaurelius.titan:titan-core:1.0.0'
compile 'org.apache.tinkerpop:gremlin-core:3.0.2-incubating'
compile 'org.apache.tinkerpop:gremlin-driver:3.0.2-incubating'
What is my goal: have fully working Graph object
What is my problem: I don't have DynamoDBStoreManager class, and I do not know what dependency I have to add.
My additional question is: why connecting through Cluster class requires only IP and works, but TitanFactory requires properties like those I have used on gremlin-server on EC2?
I do not want to create second server, I just want to connect as client to it and take Graph object.
EDIT:
After adding resolver, it builds, in output I get multiple:
13689 [TitanID(0)(4)[0]] WARN com.thinkaurelius.titan.diskstorage.idmanagement.ConsistentKeyIDAuthority - Temporary storage exception while acquiring id block - retrying in PT2.4S: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Wrote claim for id block [1, 51) in PT0.342S => too slow, threshold is: PT0.3S
and execution hangs on open() method, so does not allow me to execute any queries.
For the DynamoDBStoreManager class, you would need this dependency:
compile 'com.amazonaws:dynamodb-titan100-storage-backend:1.0.0'
Then for the DynamoDBLocal issue, try adding this resolver:
resolvers += "AWS DynamoDB Local Release Repository" at "http://dynamodb-local.s3-website-us-west-2.amazonaws.com/release"
I'm not entirely clear on what this means -- "Criteria API" instead of plain Gremlin language. I'm guessing that you mean that you want to interact with the graph using Java rather than passing Gremlin as a string over to a running Titan/Gremlin Server? If this is the case, then you don't need to start a Titan/Gremlin Server at all (step 4 above). Write an AWS Lambda program (step 2-3 above) that creates a direct Titan client connection via TitanFactory, where all of the Titan configuration properties are for your DynamoDB instance (step 5 above).
I've setup a new salt master and am trying to automate the deployment of new VMs with static IPs (no dhcp available) from a template.
I can deploy VMs ok using my template via a cloud profile with a default IP defined there, but I can't find a way to overwrite the IP address to use dynamically on deployment, I was hoping to pass the hostname/ip into the cli call or via the salt-api so I can initiate from an other application.
I've tried to pass the IP into a state as dynamic pillar data, this configures the vm hostname ok but couldn't see how to pass the IP into the profile as the profile conf doesn't accept pillar variables.
salt-call state.apply vm-new pillar='{"hostname": "salt-test", "ip": "172.0.0.11"}'
vm-new.sls
{{ pillar['hostname'] }}:
cloud.profile:
- name: {{ pillar['hostname'] }}
- profile: centos7
cloud.profiles.d/centos7.conf
...
devices:
network:
Network adapter 1:
name: 'VM Network'
switch_type: standard
ip: 172.0.0.90
subnet_mask: 255.255.255.0
gateway: [172.0.0.1]
...
I then tried to look at using a map file but trying to pass pillar data doesn't seem to work.
# salt-cloud -m cloud.maps.d/centos7.map pillar='{"hostname": 'salt-test', "ip": "172.0.0.11"}'
[ERROR ] Rendering exception occurred: Jinja variable 'salt.utils.context.NamespacedDictWrapper object' has no attribute 'hostname'
[ERROR ] Rendering map cloud.maps.d/centos7.map failed, render error:
Jinja variable 'salt.utils.context.NamespacedDictWrapper object' has no attribute 'hostname'
No nodes defined in this map
centos7.map
centos7:
- {{ pillar['hostname'] }}:
devices:
network:
Network adapter 1:
ip: {{ pillar['ip'] }}
I have spent a while digging around the docs and github issues but couple of people trying to do similar things but hardcoded IPs in the map file solved their issue, is it possible to do what I'm trying to do? Any advice/pointers on where to look next?
I've encountered a similar requirement wherein I needed to dynamically set some EC2 instance attributes (e.g. hostname). At least 3 months ago since this writing, this use case was not possible so I ended up building a salt exec module (e.g. execmodule.provision_instances) that dynamically generates a map file given my predefined profiles with default values, and eventually called salt.cloud.CloudClient.map_run with the generated map file.
It worked well by calling the exec module (e.g. salt-call execmodule.provision_instances). It would be better if we can just simply pass pillars instead of specifying a map file.
Note: Since this thread is old, salt cloud maps may already support passing pillars to map runs, please check.
I have tested salt.modules.win_ip.set_static_ip for Windows VMS and it works. For example, you can run this command on salt master to set IP of all windows machines:
salt -G 'os_family:Windows' ip.set_static_ip 'Local Area Connection' 10.1.2.3/24 gateway=10.1.2.1
You can read the official doc here.