We've been getting started with CubeJS. We are using BiqQuery, with the following heirarchy:
Project (All client)
Dataset (Corresponding to a single client)
Tables (Different data-types for a single client)
We'd like to use COMPILE_CONTEXT to allow different clients to access different Datasets based on the JWT that we issue them after authentication. The JWT includes the user info that'd cause our schema to select a different dataset:
const {
securityContext: { dataset_id },
} = COMPILE_CONTEXT;
cube(`Sessions`, {
sql: `SELECT * FROM ${ dataset_id }.sessions_export`,
measures: {
// Count of all session objects
count: {
sql: `Status`,
type: `count`,
},
In testing, we've found that the COMPILE_CONTEXT global variable is set when the server is launched, meaning that even if a different client submits a request to Cube with a different dataset_id, the old one is used by the server, sending info from the old dataset. The Cube docs on Multi-tenancy state that COMPILE_CONTEXT should be used in our scenario (at least, this is my understanding):
Multitenant COMPILE_CONTEXT should be used when users in fact access different databases. For example, if you provide SaaS ecommerce hosting and each of your customers have a separate database, then each ecommerce store should be modelled as a separate tenant.
SECURITY_CONTEXT, on the other hand, is set at Query time, so we tried to also access the appropriate data from SECURITY_CONTEXT like so:
cube(`Sessions`, {
sql: `SELECT * FROM ${SECURITY_CONTEXT.dataset_id}.sessions_export`,
But the query being sent to the database (found in the error log in the Cube dev server) is SELECT * FROM [object Object].sessions_export) AS sessions.
I'd love to inspect the SECURITY_CONTEXT variable but I'm having trouble finding how to do this, as it's only accessible within our cube Sql to my knowledge.
Any help would be appreciated! We are open to other routes besides those described above. In a nutshell, how can we deliver a specific dataset to a client using a unique JWT?
Given that all your datasets are in the same BigQuery database, I think your use-case reflects the Multiple DB Instances with Same Schema part of the documentation (that title could definitely be improved):
// cube.js
const PostgresDriver = require('#cubejs-backend/postgres-driver');
module.exports = {
contextToAppId: ({ securityContext }) =>
`CUBEJS_APP_${securityContext.dataset_id}`,
driverFactory: ({ securityContext }) =>
new PostgresDriver({
database: `${securityContext.dataset_id}`,
}),
};
// schema/Sessions.js
cube(`Sessions`, {
sql: `SELECT * FROM sessions_export`,
}
Related
We are working for FHIR(Fast Healthcare Interoperability Resources).
We have followed “FHIR works on AWS” and deployed the CloudFormation template given by AWS in our AWS environment.Following is the template that we have deployed
https://docs.aws.amazon.com/solutions/latest/fhir-works-on-aws/aws-cloudformation-template.html
Requirement : we want to maintain client specific/customized ids as primary key in the server.
Problem : server not allowing us to override or mainain client specific (customized ) ids as primary key .Infact , in the runtime, it is generating its own ids and ignoring the id given by us.
The FHIR spec allows for you to define your own IDs when using "update as create". This is when you create a new resource in the server, but use a PUT (update) request to the ID you want to create, such as Patient/1, instead of a POST (create) request to the resource URL. The server should return a 201 Created status instead of 200 OK. For more information see https://hl7.org/fhir/http.html#upsert
Not every FHIR server supports this, but if AWS does this is likely how it would work. The field in the CapabilityStatement for this feature is CapabilityStatement.rest.resource.updateCreate
EDIT:
This is possible by modifying the parameters passed to the DynamoDbDataService constructor in the deployment repo's src/config.ts
By default supportUpdateCreate, the second parameter, is set to false
const dynamoDbDataService = new DynamoDbDataService(DynamoDb, false, { enableMultiTenancy });
but you can set it to true to enable this functionality
const dynamoDbDataService = new DynamoDbDataService(DynamoDb, true, { enableMultiTenancy });
I have a developed a PowerBI report in PowerBi desktop, which is sourcing it's data from an on-prem Analysis Server.
In the PowerBi service I have created a workspace, I have then installed a on-prem ( enterprise ) gateway, with a datasource which points to the same Analysis server,then I publish my report from powerbi desktop which it then publishes into the workspace and automatically it is wired up to the gateway and all works good and well.
I want to take this pbix file and upload to a the production workspace which has a different gateway and a datasource which points to our production analysis server.
So I'm doing the following:
Uploading the Pbix to the production workspace using the powerbi powershell commandlet: New-PowerBIReport
This goes fine of course it's not wired up to anything which is expected.
I then trying to switch the gateway on the dataset for the report using this rest API call
https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/bindtogatewayingroup
Specifying the GatewayID I want it to bind to and the Datasource Id of the production analysis server.
Which then comes back with a error saying: DMTS_CanNotFindMatchingDatasourceInGatewayError
Which kind of makes sense because the report will be looking at the dev analysis server and that won't be configured on the prod gateway.
So I thought I wonder if I could change that on the report using the API, and I can by using:
https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/updatedatasourcesingroup
So using this I'm able to change the database name and server from development to Production, and verify the service has picked up this change.
Then I thought I could then call the bindtogatewayingroup and specify my production gateway and production datasource but it still fails with the same message "DMTS_CanNotFindMatchingDatasourceInGatewayError"
After hours of research I'm coming to the conclusion that it's not possible to change a report from one gateway to another, unless the exact same data source exists on both gateways, which seems a bit pointless.
Is this a limitation of the API or am I doing something wrong???
You can use connection-level parameters to define the datasource (server name and database name) and after publishing the report, to use the REST API to update their values.
Define two text parameters in your report, let's name them ServerName and DatabaseName. Set their values to point to the desired datasource. At this point when you are adding a datasource to your report, you can specify the datasource using these parameters:
If you need to amend an existing report, replace server and database names in report's M code with parameter's names. The code could look like this:
let
Source = AnalysisServices.Database(ServerName, DatabaseName, [TypedMeasureColumns=true, Implementation="2.0"]),
Model1 = Source{[Id="Model"]}[Data],
Model2 = Model1{[Id="Model"]}[Data],
#"Added Items" = Cube.Transform(Model2,
{
{Cube.AddAndExpandDimensionColumn, "[OrdersData]", {"[OrdersData].[OrderDate].[OrderDate]", "[OrdersData].[OrderYear].[OrderYear]", "[OrdersData].[Quantity].[Quantity]"}, {"OrdersData.OrderDate", "OrdersData.OrderYear"}}
})
in
#"Added Items"
Note that the first two parameters of AnalysisServices.Database are not literals, but the parameters defined above.
When you publish this report to Power BI Online, it will look for the gateway depending on the current parameter values (e.g. the one to your development server). But you can use Update Parameters In Group REST API to redirect this report to another datasource (e.g. your production server). If your gateway is properly set, you shouldn't do anything else and it will be used automatically. Probably you will want to refresh your model after that. With PowerShell you can do something like this:
Invoke-PowerBIRestMethod -Url 'groups/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/datasets/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/UpdateParameters' -Method Post -Body '{
"updateDetails": [
{
"name": "ServerName",
"newValue": "MyProductionServerName"
},
{
"name": "DatabaseName",
"newValue": "MyDatabaseName"
}
]
}'
Invoke-PowerBIRestMethod -Url 'groups/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/datasets/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/refreshes' -Method Post
Hope this helps!
Nowadays there is an API method for this type of task, allowing to alter demands for live connections to Analysis services made via data gateway:
https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/updatedatasourcesingroup#updatedatasourceconnectionrequest
call it with a body like this:
{
"updateDetails": [
{
"datasourceSelector": {
"datasourceType": "AnalysisServices",
"connectionDetails": {
"server": "server name mentioned in your pbix file",
"database": "original database name used in pbix"
}
},
"connectionDetails": {
"server": "a new server name that is used on the gateway",
"database": "potentially, another database name"
}
}
]
}
The docs (https://www.apollographql.com/docs/apollo-server/features/data-sources.html#Using-Memcached-Redis-as-a-cache-storage-backend) show code like this:
const { RedisCache } = require('apollo-server-cache-redis');
const server = new ApolloServer({
typeDefs,
resolvers,
cache: new RedisCache({
host: 'redis-server',
// Options are passed through to the Redis client
}),
dataSources: () => ({
moviesAPI: new MoviesAPI(),
}),
});
I was wondering how that cache key is used, considering it seems like the caching is actually custom implemented in something like MoviesAPI() and then used via context.dataSources.moviesAPI.someFunc(). For example, say I wanted to implement my own cache for a SQL database. It'd look like
cache: new RedisCache({
host: 'redis-server',
}),
dataSources: () => ({
SQL: new SQLCache(),
}),
});
where SQLCache has my own function that connects to the RedisCache like:
getCached(id, query, ttl) {
const cacheKey = `sqlcache:${id}`;
return redisCache.get(cacheKey).then(entry => {
if (entry) {
console.log('CACHE HIT!');
return Promise.resolve(JSON.parse(entry));
}
console.log('CACHE MISS!');
return query.then(rows => {
if (rows) redisCache.set(cacheKey, JSON.stringify(rows), ttl);
return Promise.resolve(rows);
});
});
}
So that means I have RedisCache in both the ApolloServer cache key and dataSource implementation. Clearly, the RedisCache is used in the dataSource implementation, but then what does that ApolloServer cache key do exactly?
Also on the client, examples mostly show use of InMemoryCache instead of Redis cache. Should the client Apollo cache be a different cache from the server cache or should the same cache like RedisCache be in both places?
The cache passed to the ApolloServer is, to my knowledge, strictly used in the context of a RESTDataSource. When fetching resources from the REST endpoint, the server will examine the Cache-Control header on the response, and if one exists, will cache the resource appropriately. That means if the header is max-age=86400, the response will be cached with a TTL of 24 hours, and until the cache entry expires, it will be used instead of calling the same REST url.
This is different than the caching mechanism you've implemented, since your code caches the response from the database. Their intent is the same, but they work with different resources. The only way your code would effectively duplicate what ApolloServer's cache already does is if you had written a similar DataSource for a REST endpoint instead.
While both of these caches reduce the time it takes to process your GraphQL response (fetching from cache is noticeably faster than from the database), client-side caching reduces the number of requests that have to be made to your server. Most notably, the InMemoryCache lets you reuse one query across different places in your site (like different components in React) while only fetching the query once.
Because the client-side cache is normalized, it also means if a resource is already cached when fetched through one query, you can potentially avoid refetching it when it's requested with another query. For example, if you fetch a list of Users with one query and then fetch a user with another query, your client can be configured to look for the user in the cache instead of making the second query.
It's important to note that while resources cached server-side typically have a TTL, the InMemoryCache does not. Instead, it uses "fetch policies" to determine the behavior of individual queries. This lets you, for example, have a query that always fetches from the server, regardless of what's in the cache.
Hopefully that helps to illustrate that both server-side and client-side caching are useful but in very different ways.
My application is hosted on a central server which serves many customers. It now needs to cross-reference information with the database servers that reside onsite at the customer's location.
I want to store the details of the customer's server alongside their account details (e.g dbname, host, port etc).
However depending on who is logged into the application, I need to supply their data connection details into a <cfquery> function in order to perform lookups. Something like this:
<cfquery name="rsOrders" datasource="{dynamically provided connection string}">
SELECT *
FROM
CompanysDBTable
</cfquery>
I understand that there is an Administrator API which creates a data source programmatically, however using that would mean an extra process in my system and what would happen if the data source details were to be updated by the customer?
So is there any way to do it on the fly like above? That is, supplying the data connection string within the <cfquery> tag.
Or is there a better way of doing this altogether?
You can create application-specific datasources at runtime in ColdFusion 11. See the docs: "Application-specific datasources in Application.cfc". I also discuss this in my blog: "Defining datasources in Application.cfc".
An example would be:
// Application.cfc
component {
this.name = "DSNTest02";
this.datasources = {
scratch_mssql_app = {
database = "scratch",
host = "localhost",
port = "1433",
driver = "MSSQLServer",
username = "scratch",
password = "scratch"
},
scratch_embedded_app = {
database = "C:\apps\adobe\ColdFusion\11\full\cfusion\db\scratch",
driver = "Apache Derby Embedded"
}
};
this.datasource = "scratch_mssql_app";
}
That's the closest you can get with ColdFusion.
If you were to use JDBC directly, you could just give a connection string when creating the connection, but then your code would need to contend with the record sets returned by the JDBC driver, which would not be CFML query objects.
When you get the connection credentials, set up a suitably named datasource in ColdFusion. Put the name of this datasource in the database that has the other account details you mentioned. For customers who did not send you credentials, use a default datasource.
When you run your application, get the datasource along with the other customer information and use it in your cfquery tags.
Is it possible to generate fixtures from an existing DB in Symfony2/Doctrine? How could I do that?
Example:
I have defined 15 entities and my symfony2 application is working. Now some people are able to browse to the application and by using it it had inserted about 5000 rows until now. Now I want the stuff inserted as fixtures, but I don’t want to do this by hand. How can I generate them from the DB?
There's no direct manner within Doctrine or Symfony2, but writing a code generator for it (either within or outside of sf2) would be trivial. Just pull each property and generate a line of code to set each property, then put it in your fixture loading method. Example:
<?php
$i = 0;
$entities = $em->getRepository('MyApp:Entity')->findAll();
foreach($entities as $entity)
{
$code .= "$entity_{$i} = new MyApp\Entity();\n";
$code .= "$entity_{$i}->setMyProperty('" . addslashes($entity->getMyProperty()); . "'); \n");
$code .= "$manager->persist($entity_{$i}); \n $manager->flush();";
++$i;
}
// store code somewhere with file_put_contents
As I understand your question, you have two databases: the first is already in production and filled with 5000 rows, the second one is a new database you want to use for new test and development. Is that right ?
If it is, I suggest you to create in you test environment two entity manager: the first will be the 'default' one, which will be used in your project (your controllers, etc.). The second one will be used to connect to your production database. You will find here how to deal with multiple entity manager : http://symfony.com/doc/current/cookbook/doctrine/multiple_entity_managers.html
Then, you should create a Fixture class which will have access to your container. There is an "how to" here : http://symfony.com/doc/current/bundles/DoctrineFixturesBundle/index.html#using-the-container-in-the-fixtures.
Using the container, you will have access to both entity manager. And this is the 'magic': you will have to retrieve the object from your production database, and persist them in the second entity manager, which will insert them in your test database.
I point your attention to two points:
If there are relationship between object, you will have to take care to those dependencies: owner side, inversed side, ...
If you have 5000 rows, take care on the memory your script will use. Another solution may be use native sql to retrieve all the rows from your production database and insert them in your test database. Or a SQL script...
I do not have any code to suggest to you, but I hope this idea will help you.
I assume that you want to use fixtures (and not just dump the production or staging database in the development database) because a) your schema changes and the dumps would not work if you update your code or b) you don't want to dump the hole database but only want to extend some custom fixtures. An example I can think of is: you have 206 countries in your staging database and users add cities to those countries; to keep the fixtures small you only have 5 countries in your development database, however you want to add the cities that the user added to those 5 countries in the staging database to the development database
The only solution I can think of is to use the mentioned DoctrineFixturesBundle and multiple entity managers.
First of all you should configure two database connections and two entity managers in your config.yml
doctrine:
dbal:
default_connection: default
connections:
default:
driver: %database_driver%
host: %database_host%
port: %database_port%
dbname: %database_name%
user: %database_user%
password: %database_password%
charset: UTF8
staging:
...
orm:
auto_generate_proxy_classes: %kernel.debug%
default_entity_manager: default
entity_managers:
default:
connection: default
mappings:
AcmeDemoBundle: ~
staging:
connection: staging
mappings:
AcmeDemoBundle: ~
As you can see both entity managers map the AcmeDemoBundle (in this bundle I will put the code to load the fixtures). If the second database is not on your development machine, you could just dump the SQL from the other machine to the development machine. That should be possible since we are talking about 500 rows and not about millions of rows.
What you can do next is to implement a fixture loader that uses the service container to retrieve the second entity manager and use Doctrine to query the data from the second database and save it to your development database (the default entity manager):
<?php
namespace Acme\DemoBundle\DataFixtures\ORM;
use Doctrine\Common\DataFixtures\FixtureInterface;
use Doctrine\Common\Persistence\ObjectManager;
use Symfony\Component\DependencyInjection\ContainerAwareInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Acme\DemoBundle\Entity\City;
use Acme\DemoBundle\Entity\Country;
class LoadData implements FixtureInterface, ContainerAwareInterface
{
private $container;
private $stagingManager;
public function setContainer(ContainerInterface $container = null)
{
$this->container = $container;
$this->stagingManager = $this->container->get('doctrine')->getManager('staging');
}
public function load(ObjectManager $manager)
{
$this->loadCountry($manager, 'Austria');
$this->loadCountry($manager, 'Germany');
$this->loadCountry($manager, 'France');
$this->loadCountry($manager, 'Spain');
$this->loadCountry($manager, 'Great Britain');
$manager->flush();
}
protected function loadCountry(ObjectManager $manager, $countryName)
{
$country = new Country($countryName);
$cities = $this->stagingManager->createQueryBuilder()
->select('c')
->from('AcmeDemoBundle:City', 'c')
->leftJoin('c.country', 'co')
->where('co.name = :country')
->setParameter('country', $countryName)
->getQuery()
->getResult();
foreach ($cities as $city) {
$city->setCountry($country);
$manager->persist($city);
}
$manager->persist($country);
}
}
What I did in the loadCountry method was that I load the objects from the staging entity manager, add a reference to the fixture country (the one that already exists in your current fixtures) and persist it using the default entity manager (your development database).
Sources:
DoctrineFixturesBundle
How to work with Multiple Entity Managers
you could use https://github.com/Webonaute/DoctrineFixturesGeneratorBundle
It add ability to generate fixtures for single entity using commands like
$ php bin/console doctrine:generate:fixture --entity=Blog:BlogPost --ids="12 534 124" --name="bug43" --order="1"
Or you can create full snapshot
php app/console doctrine:generate:fixture --snapshot --overwrite
The Doctrine Fixtures are useful because they allow you to create objects and insert them into the database. This is especially useful when you need to create associations or say, encode a password using one of the password encoders. If you already have the data in a database, you shouldn't really need to bring them out of that format and turn it into PHP code, only to have that PHP code insert the same data back into the database. You could probably just do an SQL dump and then re-insert them into your database again that way.
Using a fixture would make more sense if you were initiating your project but wanted to use user input to create it. If you had in your config file the default user, you could read that and insert the object.
The AliceBundle can help you doing this. Indeed it allows to load fixtures with YAML (or PHP array) files.
For instance you can define your fixtures with:
Nelmio\Entity\Group:
group1:
name: Admins
owner: '#user1->id'
Or with the same structure in a PHP array. It's WAY easier than generating working PHP code.
It also supports references:
Nelmio\Entity\User:
# ...
Nelmio\Entity\Group:
group1:
name: Admins
owner: '#user1'
In the doctrine_fixture cookbook, you can see in the last example how to get the service container in your entity.
With this service container, you can retrieve the doctrine service, then the entity manager. With the entity manager, you will be able to get all the data from your database you need.
Hope this will help you!