I am trying to add GCP labels to my dbt sql files and running into some errors.
In the BigQuery UI you can prepend any query with label key/value pairs like this:
SET ##query_label = "key1:value1,key2:value2";
I know the keys and values I am using are correct because they work in the UI.
Is there anything in how dbt parses/processes the sql that could be failing due to the ## symbols?
I tried adding SET ##query_label = "key1:value1,key2:value2"; after the config section and it fails
You can specify labels for the views and tables you create. These labels need to be provided in either the dbt_project.yml file, or within the model config. See the examples below:
# this is your dbt_project.yml
models:
your_project:
marketing:
+labels:
domain: marketing
another_key: another_value
finance:
+labels:
domain: finance
-- in the config of the model
{{
config(
materialized = "table",
labels = {'domain': 'finance'}
)
}}
select * from {{ ref('fct_revenue') }}
You can find more information in the docs here.
Related
I want to create a view in BigQuery that also shows comments such as the author, date created etc.
But if I try this in the UI the comments are left out.
Is there a way to do this in the BigQuery UI?
Or are there other ways using bq client or python, or ...?
So for example if I run this:
CREATE OR REPLACE VIEW `my_project_id.my_dataset.my_view_name`
AS
-- this is my important comment. This will be a long and extensive comment.
SELECT 1 as column_a
;
BigQuery will not show the comments in the view when the UI is used to create that view:
Bigquery is skipping the "---- this is my important comment. This will be a long and extensive comment." because it is treated as a comment on the SQL query and it is not treated as a separate string to be included in the creation of view.
Another option is to use bq command as shown below.
bq mk \
--use_legacy_sql=false \
--expiration 3600 \
--description "This is my view" \
--label organization:development \
--view \
'-- this is my important comment. This will be a long and extensive comment.
SELECT 1 as column_a ' \
your-dataset.your-view
My sample output:
I don't know how to do this in the UI, but with the python API you can do as follows:
from google.cloud import bigquery
bq_client = bigquery.Client()
view_id = "my_project_id.my_dataset.my_view_name"
view = bigquery.Table(view_id)
query = """
-- this is my important comment. This will be a long and extensive comment.
SELECT 1 as column_a
"""
view.view_use_legacy_sql = False
view.view_query = query
# if your view already exists
bq_client.delete_table(view)
# your query will now show the comment at the top
bq_client.create_table(view)
This results in the following view:
See also: https://cloud.google.com/bigquery/docs/views#python
#app.route('/view_case/<case_name>')
def view_case(case_name):
query = db.collection('cases').document(case_name).collection('documents').get()
documents = []
for _document in query:
documents.append(_document)
return render_template('views/view_case.html', documents=documents)
Is the above method the correct way to query a group of documents and send them to the DOM as a list to be iterated over by jinja to display?
Side question, I notice the results dont include the documents ID, is there a way to attach the id to the document?
Just amended to use list comprehension
from google.cloud import firestore
db = firestore.Client()
collection_ref = db.collection(u'collection').get()
documents = list(doc.to_dict() for doc in collection_ref)
I have some information related to different vendors in my database and I want to allow each registered vendor (representative person) to view slices/dashboards which contains only data related to them.
One possible solution could be to create separate views for each vendor as well as separate roles for each vendor. But it feels like a bad idea if you have 100+ vendors (as is my case); and it's not a flexible or scalable solution.
Is there some way to automatically filter a given view for each user? For example, we have a "general profit by product" bar chart, and user X can see only products of vendor X
What you're looking for is multi-tenancy support, and this is not currently supported out-of-the-box in Superset.
There is however an open PR for one possible solution: https://github.com/apache/incubator-superset/pull/3729
One option could be to re-use and/or adapt that code for your use-case.
Another option might be to look into JINJA_CONTEXT_ADDONS [https://github.com/apache/incubator-superset/blob/master/docs/installation.rst#sql-lab] and see whether you might be able to pass additional context to your query (e.g. your vendor_id) and restrict the scope of your query using that parameter.
Superset config has the below two configurations(DB_CONNECTION_MUTATOR, SQL_QUERY_MUTATOR), which can allow for multi-tenancy to an extent.
A callable that allows altering the database conneciton URL and params
on the fly, at runtime. This allows for things like impersonation or
arbitrary logic. For instance you can wire different users to
use different connection parameters, or pass their email address as the
username. The function receives the connection uri object, connection
params, the username, and returns the mutated uri and params objects.
Example:
def DB_CONNECTION_MUTATOR(uri, params, username, security_manager, source):
user = security_manager.find_user(username=username)
if user and user.email:
uri.username = user.email
return uri, params
Note that the returned uri and params are passed directly to sqlalchemy's
as such create_engine(url, **params)
DB_CONNECTION_MUTATOR = None
A function that intercepts the SQL to be executed and can alter it.
The use case is can be around adding some sort of comment header
with information such as the username and worker node information
def SQL_QUERY_MUTATOR(sql, username, security_manager):
dttm = datetime.now().isoformat()
return f"-- [SQL LAB] {username} {dttm}\n{sql}"
SQL_QUERY_MUTATOR = None
One easy way of solving this problem is by using pre-defined JINJA parameters.
Two parameters that can be used are '{{current_username() }}' and {{current_user_id() }}
First you need to ensure that you can use JINJA templates -
In superset_config.py add the following
FEATURE_FLAGS = {
"ENABLE_TEMPLATE_PROCESSING": True,
}
Restart
Now if you go to the SQL LAB and type the following -
SELECT '{{ current_username() }}',{{ current_user_id() }};
You should get an output
?column?
?column?__1
PayalC
5
Now all you have to do is append one of the two following sql snippet in all your queries.
select ........ from ...... where ...... vendorid={{ current_user_id() }}
select ........ from ...... where ...... vendorname='{{ current_username() }}'
vendorid={{ current_user_id() }} and/or
vendorname='{{ current_username() }}' will restrict the user to view only her data.
You could also make it more flexible by creating a table which has a mapping of user to vendorid. That table can be your added to all the queries and you could map multiple vendors to a single user or even all vendors to a single user for a super admin.
I'm using Django for a project. I have a .sql file containing insert statements. How can I get them into my sqlite3 database and have them work with my models?
The project "fortune" has a models.py file that looks like the following:
class fortune(models.Model):
id = models.IntegerField(primary_key=True)
category = models.CharField(max_length=50)
length = models.IntegerField()
aphorism = models.CharField(max_length=5000)
I have a .sql file with a list of Insert statements like the follwing:
INSERT INTO "fortune_fortune" VALUES(1,'fortunes',127,'Arbitrary Text');
When I run .schema on my db.sqlite3 file which is configured with my project I see:
CREATE TABLE fortune_fortune(id integer, category varchar(50), length integer, aphorism varchar(5000));
I've tried using .read in my sqlite shell with no luck. I've tried typing "sqlite3 file.sqlite3 < file.sql" in bash as well. There is something I'm missing here, but I can't seem to ID the problem.
Thanks for your help.
ok, wait.. normally you dont use sql statements to insert data into db, if you work with django.
to insert data into db, you work with django ORM which is way much fun than these ugly sql statements.
fortune = fortune(category='new cat', length=19, aphorism='i love life')
fortune.save()
then as a result, you will have one new row in fortune table in your db. just read django docs and you feel happy!
and one more thing, class names are always in capital.
to your issue:
Django provides a hook for passing the database arbitrary SQL that’s executed just after the CREATE TABLE statements when you run migrate. You can use this hook to populate default records, or you could also create SQL functions, views, triggers, etc.
The hook is simple: Django just looks for a file called sql/<modelname>.sql, in your app directory, where <modelname> is the model’s name in lowercase.
more in docs
I follow the guide to create:
1. a action of sonata admin http://sonata-project.org/bundles/admin/master/doc/cookbook/recipe_custom_action.html
2. a batch ax http://sonata-project.org/bundles/admin/master/doc/reference/batch_actions.html
In my action I want to change 1 field of the entity from true to false or via versa. For examble : the field enable. How can I update change to the database
I solve the a single action
However, with batch action when I load data from database with the code
$target = $modelManager->find($this->admin->getClass(), $request->get('idx'));
Sonata raise a notice "ModelManager ->find ('C4C\PlatformBundle\Entity\Member', array('1', '2', '3')) ". The reason is that Model Manage can not find an objec by an array.
How can I get all the objects from database based on an array