We are connecting to BigTable using HBase API and we are using the hbase-site.xml.
Is there any way we can use impersonation using HBase API to connect to BigTable?
<configuration xmlns:xi="http://www.w3.org/2001/XInclude">
<property>
<name>hbase.client.connection.impl</name>
<value>com.google.cloud.bigtable.hbase1_x.BigtableConnection</value>
</property>
<property>
<name>google.bigtable.project.id</name>
<value></value>
</property>
<property>
<name>google.bigtable.instance.id</name>
<value></value>
</property>
<property>
<name>google.bigtable.auth.json.keyfile</name>
<value></value>
</property>
</configuration>
The source code (bigtable implementation using HBase API i.e com.google.cloud.bigtable.hbase1_x.BigtableConnection)doesn't have any functionality related to using impersonation. https://github.com/googleapis/java-bigtable-hbase
About User Impersonation in Hbase, it appears that it is supported through the user of an Apache Thrift server, which I think acts a bit like an upstream proxy. Per the comments in the post here, it is stated that CBT does support thrift with this provided example (note this should be set up on a GCE instance). This additional guide shows the process of setting up this gateway and using it for requests coming from App Engine. If I misunderstood your intention, you can come back with additional details on your use-case, so that I could work on your question.
We didn't anyway to configure the impersonated user in hbase-site.xml as in the source code of this didn't find any param for this https://github.com/googleapis/java-bigtable-hbase
The best way, we can impersonate using HBase API when connecting to BigTable is create BigTable connection using impersonation and use that connection object in the existing HBase API implementation. Here is the code snippet for getting the connection
public org.apache.hadoop.hbase.client.Connection getConnection() throws Exception{
Credentials credentials = GoogleCredentials.fromStream(new FileInputStream("credentials_key.json"));
ImpersonatedCredentials targetCredentials = ImpersonatedCredentials.create((GoogleCredentials) credentials,
"your-service-account#gcp-test-project.iam.gserviceaccount.com", null,
Arrays.asList("https://www.googleapis.com/auth/bigtable.data"), 3600);
// use your gcp project name and bigtable instance name
Configuration config = BigtableConfiguration.configure("gcp-test-project", "big-table-instance");
BigtableConfiguration.withCredentials(config,(Credentials)targetCredentials);
Connection connection = BigtableConfiguration.connect(config);
return connection;
}
Using this approach, with minimal changes, one can use existing api/implementations using HBase API to connect to BigTable and can impersonate. Please note that if impersonation is not required (the json key account you are using has the permissions to read/write), then there will not be any changes required for your existing code base. ref https://cloud.google.com/bigtable/docs/hbase-bigtable
Related
I am trying to use WSO2 to schedule pooling data call every minutes to a REST API my business has and push that information to our centralize MQTT broker.
I've been reading the documents of the Streaming Integrator, Micro Integrator, Micro Gateway and API Manager and I cannot find any way to schedule REST API calls base on a defined time.
The point of this task is to push data from all our system into our centralize broker and add analyzing tool afterward to benefit from the data created by our systems that is only accessible by the system at this time.
Could someone give me a hint on what should be the right tools for this and maybe the link to some documentation about how to configure time base call if the software wso2 allowed it ?
You can create a WSO2EI scheduled task
You can define a cron job expression for timing and execute a sequence or an implementation class.
example:
<task name="SampleInjectToSequenceTask"
class="org.apache.synapse.startup.tasks.MessageInjector"
group="synapse.simple.quartz">
<trigger interval="5"/>
<property xmlns:task="http://www.wso2.org/products/wso2commons/tasks"
name="injectTo"
value="sequence"/>
<property xmlns:task="http://www.wso2.org/products/wso2commons/tasks"
name="sequenceName"
value="SampleSequence"/>
</task>
I'm currently developing a REST service to replace an existing solution. I'm using plain Payara/JEE7/JAX-RS. I am not using Spring and I do not intent to.
The problem I'm facing is that we want to reuse as much of the original configuration as possible (deployment on multiple nodes in a cluster with puppet controlling the configuration files).
Usually in Glassfish/Payara, you'd have a domain.xml file that has some content like this:
<jdbc-connection-pool driver-classname="" pool-resize-quantity="10" datasource-classname="org.postgresql.ds.PGSimpleDataSource" max-pool-size="20" res-type="javax.sql.DataSource" steady-pool-size="10" description="" name="pgsqlPool">
<property name="User" value="some_user"/>
<property name="DatabaseName" value="myDatabase"/>
<property name="LogLevel" value="0"/>
<property name="Password" value="some_password"/>
<!-- bla --->
</jdbc-connection-pool>
<jdbc-resource pool-name="pgsqlPool" description="" jndi-name="jdbc/pgsql"/>
Additionally you'd have a persistence.xml file in your archive like this:
<persistence-unit name="myDatabase">
<provider>org.hibernate.ejb.HibernatePersistence</provider>
<jta-data-source>jdbc/pgsql</jta-data-source>
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.PostgreSQLDialect"/>
<!-- bla -->
</properties>
</persistence-unit>
I need to replace both of these configuration files by a programmatic solution so I can read from the existing legacy configuration files and (if needed) create the connection pools and persistence units on the server's startup.
Do you have any idea how to accomplish that?
Actually you do not need to edit each domain.xml by hands. Just create glassfish-resources.xml file like this:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE resources PUBLIC "-//GlassFish.org//DTD GlassFish Application Server 3.1 Resource Definitions//EN" "http://glassfish.org/dtds/glassfish-resources_1_5.dtd">
<resources>
<jdbc-connection-pool driver-classname="" pool-resize-quantity="10" datasource-classname="org.postgresql.ds.PGSimpleDataSource" max-pool-size="20" res-type="javax.sql.DataSource" steady-pool-size="10" description="" name="pgsqlPool">
<property name="User" value="some_user"/>
<property name="DatabaseName" value="myDatabase"/>
<property name="LogLevel" value="0"/>
<property name="Password" value="some_password"/>
<!-- bla --->
</jdbc-connection-pool>
<jdbc-resource pool-name="pgsqlPool" description="" jndi-name="jdbc/pgsql"/>
</resources>
Then either use
$PAYARA_HOME/bin/asadmin add-resources glassfish-resources.xml
on each node once or put it under WEB-INF/ of your war (note, in this case jndi-name SHOULD be java:app/jdbc/pgsql because you do not have access to global: scope at this context).
Note that your persistence.xml should be under META-INF/ of any jar in your classpath.
If you do not like this, you may use
#PersistenceUnit(unitName = "MyDatabase")
EmtityManagerFactory emf;
to create EntityManager on fly:
createEntityManager(java.util.Map properties).
By the way, using Payara you can share configuration with JCache across you cluster.
Since the goal is to have a dockerized server that runs a single application, I can very well use an embedded server.
Using an embedded sever, the solution to my problem looks roughly like this:
For the server project, create a Maven dependency:
<dependencies>
<dependency>
<groupId>fish.payara.extras</groupId>
<artifactId>payara-embedded-all</artifactId>
<version>4.1.1.163.0.1</version>
</dependency>
</dependencies>
Start your server like this:
final BootstrapProperties bootstrapProperties = new BootstrapProperties();
final GlassFishRuntime runtime = GlassFishRuntime.bootstrap();
final GlassFishProperties glassfishProperties = new GlassFishProperties();
final GlassFish glassfish = runtime.newGlassFish(glassfishProperties);
glassfish.start();
Add your connection pools to the started instance:
final CommandResult createPoolCommandResult = commandRunner.run("create-jdbc-connection-pool",
"--datasourceclassname=org.postgresql.ds.PGConnectionPoolDataSource", "--restype=javax.sql.ConnectionPoolDataSource", //
"--property=DatabaseName=mydb"//
+ ":ServerName=127.0.0.1"//
+ ":PortNumber=5432"//
+ ":User=myUser"//
+ ":Password=myPassword"//
//other properties
, "Mydb"); //the pool name
Add a corresponding jdbc resource:
final CommandResult createResourceCommandResult = commandRunner.run("create-jdbc-resource", "--connectionpoolid=Mydb", "jdbc__Mydb");
(In the real world you would get the data from some external configuration file)
Now deploy your application:
glassfish.getDeployer().deploy(new File(pathToWarFile));
(Usually you would read your applications from some deployment directory)
In the application itself you can just refer to the configured pools like this:
#PersistenceContext(unitName = "mydb")
EntityManager mydbEm;
Done.
A glassfish-resources.xml would have been possible too, but with a catch: My configuration file is external, shared by some applications (so the file format is not mine) and created by external tools on deployment. I would need to XSLT the file to a glassfish-resources.xml file and run a script that does the "asadmin" calls.
Running an embedded server is an all-java solution that I can easily build on a CI server and my application's test suite could spin up the same embedded server build to run some integration tests.
Is the Azure Service Fabric Reverse Proxy available in an on-premises cluster? If so, how can I enable it for an existing cluster?
The Service Fabric Reverse Proxy is described here. It allows clients external to the cluster to access application services by name with a special URL, without needing to know the exact host:port on which an instance of the service is running (which may change as services are automatically moved around).
By default the Service Fabric Reverse Proxy does not appear to be enabled for my on-prem cluster with two instances of a stateless service. I tried using the documented port 19008 but could not reach the service using the recommended URI syntax.
To wit, this works:
http://fqdn:20001/api/odata/v1/$metadata
but this does not:
http://fqdn:19008/MyApp/MyService/api/odata/v1/$metadata
In the NodeTypes section of the ClusterConfig JSON used to set up my on-prem cluster, there is a property "httpGatewayEndpointPort": "19080", but that port does not appear to work as a reverse proxy (it is the Service Fabric Explorer web-app endpoint). I am guessing that the needed configuration is specified somehow in the cluster config JSON. There are instructions in the referenced article that explain how to configure the reverse proxy in the cloud, but not on-premises.
What I am looking for are instructions on how to set up the Service Fabric reverse proxy in an on-premises multi-machine cluster or dev cluster.
Yes, the reverse proxy is available on-premises.
To get it working for an existing cluster, it must be configured and enabled in the cluster config XML and then the new config must be deployed, as described below.
For a new cluster, set it up in the cluster config JSON before creating the cluster, as described by #Scott Weldon.
#Senj provided the clue (thanks!) that led me to the answer. I had recently updated my Service Fabric bits on my dev box to 5.1.163.9590. When I looked in C:\SfDevCluster\Data\FabricHostSettings.xml, I noticed the following:
<Section Name="FabricNode">
...
<Parameter Name="NodeVersion" Value="5.1.163.9590:1.0:0" />
...
<Parameter Name="HttpApplicationGatewayListenAddress" Value="19081" />
<Parameter Name="HttpApplicationGatewayProtocol" Value="http" />
...
</Section>
Interesting! With the dev cluster fired up, I browsed to:
http://localhost:19081/MyApp/MyService/api/odata/v1/$metadata
and voila! My API returned the expected data. So #Senj was correct that it has to do with the HttpApplicationGateway settings. I am guessing that in the latest SDK version it is pre-configured and enabled by default. (What threw me off is all the docs refer to port 19008, but the actual configured port was 19081!)
In order to get the reverse proxy to work on the 'real' multi-machine (VM) cluster, I did the following (Note: I don't think upgrading the cluster codepackage was necessary, but since I had nothing in my image store for the cluster upgrade, and the cluster upgrade process requires a code package, I used the latest version):
Copy the existing cluster manifest (from the Manifest tab in Service Fabric Explorer), paste into a new XML file, bump the version number and modify as follows:
To the NodeType Endpoints section, add:
<NodeTypes>
<NodeType Name="NodeType0">
<Endpoints>
<HttpApplicationGatewayEndpoint Port="19081" Protocol="http" />
...
</Endpoints>
</NodeType>
</NodeTypes>
and under <FabricSettings>, add the following section:
<Section Name="ApplicationGateway/Http">
<Parameter Name="IsEnabled" Value="true" />
</Section>
Using Service Fabric PowerShell commands:
Copy the new cluster config (the previously copied manifest.xml) to the fabric image store
Register the new cluster config
Copy the Service Fabric Runtime cluster codepackage (available here - see the release notes for the link to the MSI) to the image store
Register the cluster codepackage
Start and complete cluster upgrade (I used unmonitored manual mode, which does one VM at a time and requires a manual Resume command after each node is complete)
After the cluster upgrade was complete, I was able to query my service API using the reverse proxy endpoint and appname/servicename URL syntax:
http://fqdn:19081/MyApp/MyService/api/odata/v1/$metadata
I enabled this in the standalone installer version (5.1.156) by adding the following line to the JSON configuration file under the nodeTypes element (I used ClusterConfig.Unsecure.MultiMachine.json but I assume any of the JSON files would work):
"httpApplicationGatewayEndpointPort": "19081"
So the final nodeTypes looked like this:
"nodeTypes": [
{
"name": "NodeType0",
"clientConnectionEndpointPort": "19000",
"clusterConnectionEndpoint": "19001",
"httpGatewayEndpointPort": "19080",
"httpApplicationGatewayEndpointPort": "19081",
"applicationPorts": {
"startPort": "20001",
"endPort": "20031"
},
"ephemeralPorts": {
"startPort": "20032",
"endPort": "20062"
},
"isPrimary": true
}
]
I think it has something to do with the HttpApplicationGatewayEndpoint property, see also my question on https://github.com/Azure/service-fabric-issues/issues/5
But it doesn't work for me..
Also notice that
<Section Name="ApplicationGateway/Http">
<Parameter Name="IsEnabled" Value="true" />
</Section>
is true for me.
Edit:
I noticed that on my Windows-Only installation, HttpApplicationGatewayListenAddress has value 0 in the FabricHostSettings.xml
<Parameter Name="HttpGatewayListenAddress" Value="19080" />
<Parameter Name="HttpGatewayProtocol" Value="http" />
<Parameter Name="HttpApplicationGatewayListenAddress" Value="0" />
<Parameter Name="HttpApplicationGatewayProtocol" Value="" />
Just installed Windows Server AppFabric 1.1 on my Windows 7 box and I'm trying to write some code in a console application against the cache client API as part of an evaluation of AppFabric. My app.config looks like the following:
<configSections>
<section name="dataCacheClient" type="Microsoft.ApplicationServer.Caching.DataCacheClientSection, Microsoft.ApplicationServer.Caching.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" allowLocation="true" allowDefinition="Everywhere" />
</configSections>
<dataCacheClient>
<hosts>
<host name="pa-chouse2" cachePort="22233" />
</hosts>
</dataCacheClient>
I created a new cache and added my domain user account as an allowed client account using the Powershell cmdlet Grant-CacheAllowedClientAccount. I'm creating a new DataCache instance like so:
using (DataCacheFactory cacheFactory = new DataCacheFactory())
{
this.cache = cacheFactory.GetDefaultCache();
}
When I call DataCache.Get, I end up with the following exception:
ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)
I'd be very grateful if anyone could point out what I'm missing to get this working.
Finally figured out what my problem was after nearly ripping out what little hair I have left. Initially, I was creating my DataCache like so:
using (DataCacheFactory cacheFactory = new DataCacheFactory())
{
this.cache = cacheFactory.GetDefaultCache();
}
It turns out that DataCache didn't like having the DataCacheFactory created it disposed of. Once I refactored the code so that my DataCacheFactory stayed in scope as long as I needed my DataCache, it worked as expected.
What does your Spring configuration for integration tests look like using an embedded h2 datasource and, optionally, JUnit?
My first try with a SingleConnectionDataSource basically worked, but failed on more complicated tests where you need several connections at the same time or suspended transactions. I think h2 in tcp based server mode might work as well, but this is probably not the fastest communication mode for a temporary embedded database in memory.
What are the possibilities and their advantages / disadvantages? Also, how do you create the tables / populate the database?
Update: Let's specify some concrete requirements that are important for such tests.
The database should be temporary and in memory
The connection should probably not use tcp, for speed requirements
It would be nice if I could use a database tool to inspect the content of the database during debugging
We have to define a datasource since we can't use the application servers datasource in unit tests
With the reservation that I do not know if there is any tool that can inspect the database, I think that a simple solution would be to use the Spring embedded database (3.1.x docs, current docs) which supports HSQL, H2, and Derby.
Using H2, your xml configuration would look like the following:
<jdbc:embedded-database id="dataSource" type="H2">
<jdbc:script location="classpath:db-schema.sql"/>
<jdbc:script location="classpath:db-test-data.sql"/>
</jdbc:embedded-database>
If you prefer Java based configuration, you can instantiate a DataSource like this (note that EmbeddedDataBase extends DataSource):
#Bean(destroyMethod = "shutdown")
public EmbeddedDatabase dataSource() {
return new EmbeddedDatabaseBuilder().
setType(EmbeddedDatabaseType.H2).
addScript("db-schema.sql").
addScript("db-test-data.sql").
build();
}
The database tables are created by the db-schema.sql script and they are populated with test data from the db-test-data.sql script.
Don't forget to add the H2 database driver to your classpath.
I currently include in a test-only springconfig-file as a datasource:
<bean id="database.dataSource" class="org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy">
<constructor-arg>
<bean class="org.springframework.jdbc.datasource.SimpleDriverDataSource">
<property name="driverClass" value="org.h2.Driver" />
<property name="url"
value="jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;MODE=Oracle;TRACE_LEVEL_SYSTEM_OUT=2" />
</bean>
</constructor-arg>
</bean>
<!-- provides a H2 console to look into the db if necessary -->
<bean id="org.h2.tools.Server-WebServer" class="org.h2.tools.Server"
factory-method="createWebServer" depends-on="database.dataSource"
init-method="start" lazy-init="false">
<constructor-arg value="-web,-webPort,11111" />
</bean>
Creating / dropping the tables can be done by using executeSqlScript when overriding AbstractAnnotationAwareTransactionalTests.onSetUpBeforeTransaction, or with SimpleJdbcTestUtils.executeSqlScript in an appropriate place.
Compare also this posting.
H2 is bundled with a built-in connection pool implementation. The following XML provides an example of using it as a Datasource bean without a need to introduce additional dependencies on DBCP or C3P0:
<bean id="dataSource" class="org.h2.jdbcx.JdbcConnectionPool" destroy-method="dispose">
<constructor-arg>
<bean class="org.h2.jdbcx.JdbcDataSource">
<property name="URL" value="jdbc:h2:dbname"/>
<property name="user" value="user"/>
<property name="password" value="password"/>
</bean>
</constructor-arg>
</bean>
The database will be shut down by calling a dispose method when Spring application context closes.
I think it's best to use your production DataSource implementation (only with different connection-string) for the unit-tests.
Anyway "failed on more complicated tests" doesn't give enough information for a more detailed answer.
(Self-ad : check this)