I have a main table with tasks and the date completed. I wanted to put the completion dates into quarter buckets. instead of building complicated if statements, I am thinking of having a separate dynamic table. Is it possible to map the date completed in table1 into the respective buckets when it falls within the start and end dates of the quarters? if yes, how
Related
Yesterday I scheduled daily the overwriting of a table. The new table will be partitioned as well as the overwritten one... It did not run at the corresponding time, nor gave an error... It just did not started.
My feeling is that it has to be with the partitioning option. To mention that the casting of the field date_formatted that will be used as partition field works fine.
As far as I know, when scheduling a query you can't use the create or replace table T partitioned by column C as select...
You starts from the select... clause, as shows in the image, and I don't know if the problem comes from here.
PS: I had no troubles scheduling the appending to a partitioned by day table with this same procedure.
the destination table is in the same dataset.
if the very same query is scheduled to deliver the results in a table with the same name, but in a different dataset (located in the same project), it works.
by the way, the input table and the output table never were the same.
I'm trying to figure out the best way to build a relationship from a table that has records in a daily format (one record represents a single date) to a table that contains records in a date-range format (one record has a start-date and end-date, consequentially representing a period or range of dates).
Since my actual datafiles contain work-related information, I created 2 demo tables that contain dummy data that reflects the date columns in question.
Here is my DailyDate table
Here is my DateRanges table
Here is the current model view
I would like to be able to have a relationship built between the tables so that if I were to have 2 tables/matrices in the Report view, with one table showing the Daily Date Data and the other table showing the Ranged Data Data, I would be able to select a record in the Daily table and Power BI's highlight functionality would filter the records in the Ranged table so that only date ranges containing my selected date appear, and vice-versa (if possible).
For example, referencing this screenshot, if I were to select index 0 in the 'Daily Data Data' table, the 'Ranged Date Data' table should be filtered to only show records with index of 0 in the other table. If I were to select Index 2 (01/03/2022) in the Daily Date Data table then the Ranged Date Data table should be filtered to only show indices (0, 1).
In the model view, when trying to build this relationship, I can create a relationship from DailyDates.Date to DateRanges.StartDate and then from DailyDates.Date to DateRanges.EndDate; however, only a single relationship can be active so the highlight and slicer functionality will not give me the results I'm looking for.
As you can see from this demo, the datasets are small; however, my actual datasets contain around 50 million records in the Daily table and 10+ million records in teh Ranged table, so I'm hoping there can be an efficient method of getting this functionality that will not be too much of a load on memory.
Any advice into how to accomplish this would be greatly appreciated.
I need to create a measure with a logical DAX function that will iterate row by row between three tables and it will return me a virtual table of the rows that are in the first table and not in the other two.
My goal is to create a Maintenance Dashboard and I need to show how many activities were canceled.
The data is organized by the following:
Table 1 is dAcitivities0, this table is the first planning that happened in the begninning of the year.
Table 2 is the fReporgrammed, it shows me all the reprogrammed activities (dates) and it has the ID from dActivities0;
Table 3 is the fExecuted, it shows me all the activities that were executed and it is also filtered by dAcitivities with its ID.
Let there be an external table in Athena which points to a large amount of data stored in parquet format on s3. It contains a lot of columns and is partitioned on a field called 'timeid'. Now, there's another external table (small one) which maps timeid to date.
When the smaller table is also partitioned on timeid and we join them on their partition id (timeid) and put date into where clause, only those specific records are scanned from large table which contain timeids corresponding to that date. The entire data is not scanned here.
However, if the smaller table is not partitioned on timeid, full data scan takes place even in the presence of condition on date column.
Is there a way to avoid full data scan even when the large partitioned table is joined with an unpartitioned small table? This is required because the small table contains only one record per timeid and it might not be expected to create a separate file for each.
That's an interesting discovery!
You might be able to avoid the large scan by using a sub-query instead of a join.
Instead of:
SELECT ...
FROM large-table
JOIN small-table
WHERE small-table.date > '2017-08-03'
you might be able to use:
SELECT ...
FROM large-table
WHERE large-table.date IN
(SELECT date from small-table
WHERE date > '2017-08-03')
I haven't tested it, but that would avoid the JOIN you mention.
I'm attempting to create a shared date dimensions between two fact tables in Power BI, based off of a relational data source.
Currently, if I include an unrelated dimension in the report, I get numbers duplicated across multiple rows, where they don't really apply.
I'm wondering if there is any way to tell Power BI that certain dimensions cannot be used with certain fact tables, similar to using IgnoreUnrelatedDimensions in SSAS.
Currently the only solution I can find is to create a separate date dimension, so that the two fact tables have no relationship that could be used to join them, however this would mean forfeiting the ability to do any time based comparisons.
Create a combined view of the fact tables with only compatible columns to be used for time based comparison:
In Query Editor, create new queries for your fact tables by
referencing i.e. right click original query and select "Reference".
Then in those "copies" cut out the incompatible dimensions.
Rename columns to align terminology (e.g. Sales Date ==> Transaction Date, Payment Date ==> Transaction Date).
Use "Merge Queries" function to combine the copies using Full Outer Join.
Join this merged view to your date dimension