I've my data in the Table Storage from Storage Account.
In PowerBI, I want to display them without copying all data into the PowerBI.
How can i create something like a liveConnection, that every filtering option from PowerBI is queried against the Table Storage from Storage Account and retrieve the data according to my query?
Related
I am trying to copy a table in the US region to the EU region. I tried Google cloud data transfer service but it copies at the dataset level.
My dataset has lots of huge tables and I just need to copy one of them daily.
Question 1: Does google cloud data transfer service works for a table, not a dataset?
Question 2: What is the easiest solution to do the cross-region transfer for a table?
This is my dataset copy job using google cloud data transfer service: At the dataset level, it works fine but does not allow me to add the table name.
That is happening because the source type is chosen: Dataset Copy which means copies the whole dataset. There is no other option in the source type drop-down list for table transfer though.
I'm did ETL for our data and did simple aggregations on it in Athena. Our plan is to use our BI tool to access those tables from Athena. It works for now, but I'm worried that these tables are static i.e. they only reflect the data since I last created the Athena table. When called, are Athena tables automatically ran again? If not, how do I make them be automatically updated when called by our BI tool?
My only solution thus far to overwrite the tables we have is by running two different queries: one query to drop the table, and another to re-create the table. Since it's two different queries, I'm not sure if you can run it all at the same time (at least in Athena, you can't run them all in one go).
Amazon Athena is a query engine, not a database.
When a query is sent to Amazon Athena, it looks at the location stored in the table's DDL. Athena then goes to the Amazon S3 location specified and scans the files for the requested data.
Therefore, every Athena query always reflects the data shown in the underlying Amazon S3 objects:
Want to add data to a table? Then store an additional object in that location.
Want to delete data from a table? Then delete the underlying object that contains that data.
There is no need to "drop a table, then re-create the table". The table will always reflect the current data stored in Amazon S3. In fact, the table doesn't actually exist -- rather, it is simply a definition of what the table should contain and where to find the data in S3.
The best use-case for Athena is querying large quantities of rarely-accessed data stored in Amazon S3. If the data is often accessed and updated, then a traditional database or data warehouse (eg Amazon Redshift) would be more appropriate.
Pointing a Business Intelligence tool to Athena is quite acceptable, but you need to have proper processes in place for updating the underlying data in Amazon S3.
I would also recommend storing the data in Snappy-compressed Parquet files, which will make Athena queries faster and lower cost (because it is charged based upon the amount of data read from disk).
I have a use case where I need to sync spanner table with Big Query tables. So I need to update the Spanner tables based on the updated data in Big Query tables. I am planning to using Cloud data fusion for this. But I do not see any example available for this scenario. Any pointers on this?
When we create a data flow inside a power bi workspace, there are 2 options where the data flow data gets stored:
Power BI managed data lake
Bring your own azure data lake
In the 1st case, is the shared storage of the workspace impacted by the data that the dataflow puts into the power bi managed data lake? If not then what is the limit of the power bi managed data lake storage per workspace?
I have created the Big Query out of the data I have in my Cloud storage bucket.
In my use case, I am sending data periodically to the same bucket which is backend of my Big Query(while creating the Big query table I used the same bucket name).
Is it possible to get the updated data into Big Query, as I am pushing new data each time into the same bucket on some interval basis.
Just to mention - I am making native Big query from my dedicated storage bucket mentioned above.
Your help will be much appreciated. thanks in advance.
You can create an external (federated) table on Google Cloud Storage Bucket. In this case, whenever you query this table you will get the latest data.
If you just need to append data to a table (let's call it target table) based on data from the bucket - I can imagine following this process:
Create a federated table on the GCS bucket
Setup a simple cron job that runs a bq command which is just doing select * from [federated_table] and appends results into the target table (you may have a more complicated query that will check duplication of data in the target table and only appends new data).
Alternative option:
Setup a trigger on your bucket that activates cloud function and in a cloud function you just load the newly added data to the target table.