I am facing a challenge right now.
I need to combine various CSV into one database. The "catch" is:
The files have the columns as rows;
The files may have different number of rows (columns).
Here is an example below:
File1
Agent Name,xpto1
Agent Email,xpto1#abcd.com
Date,04/18/2019 14:58:25
Time Zone,Europe/Lisbon
Time Filter Begin,04/17/2019 00:00:00
Time Filter End,04/17/2019 23:00:00
Total Unavailable Time (hh:mm:ss),00:00:00
Total Offline Time (hh:mm:ss),00:47:50
File2
Agent Name,xpto2
Agent Email,xpto2#abcd.com
Date,04/18/2019 14:58:25
Time Zone,Europe/Lisbon
Time Filter Begin,04/17/2019 00:00:00
Time Filter End,04/17/2019 23:00:00
Total Auto Answered Calls Notifications Offered,27
Total Number of Calls Notification Offered,42
Total Auto Answered Calls Connected,15
Total Number of Calls Connected,15
Total Number of Multicast Call Notifications Offered,15
Total Unavailable Time (hh:mm:ss),00:02:32
Total Offline Time (hh:mm:ss),14:54:48
Total Wrap-up Time (hh:mm:ss),01:40:37
Total Email Time (hh:mm:ss),00:42:50
Total In-call Time (hh:mm:ss),03:57:42
Total Available Time (hh:mm:ss),01:04:56
Total Meal Time (hh:mm:ss),01:00:59
Total Break Time (hh:mm:ss),00:35:39
Basically I need to combine a hundred (give or take) of these files in a database format
Column 1, Column2, etc, with the correspondent values below each column, with nulls/zeros in case some file does not have a row that other may have.
Any clue how to do this?
I've tried to pivot all the files after merging but I am getting an error:
"Expression.Error: There were too many elements in the enumeration to complete the operation.
Details:
List"
Thanks in advance!
If you have all of these files in a folder, then you can connect to that folder.
If you click that double down arrow in the Content column, then it should automatically do a bunch of steps to get you to a table looking like this:
From here, you want to pivot Column1 using Column2 as the Values Column and Don't Aggregate under Advanced options. This should give you what you're looking for:
Related
Please, I created my table using hour time partition. Please, I would like would like to fetch data that was stored in my table in the last X minutes, eg last 5 minutes.
I tried using this command
SELECT *
FROM mydataset.mytable
FOR SYSTEM_TIME AS OF TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 5 MINUTE);
But, it returns a lot more rows than what is expected. I typically store 500 rows every 2 minutes, but this query is returning more than 30000 rows
As #Samuel mentioned in the comment, below example query can be considered to fetch data for a specific time period in BigQuery.
Select * from `dataset.table`
WHERE col_timestamp < TIMESTAMP_SUB(CURRENT_TIMESTAMP(),
INTERVAL 5 MINUTE)
Posting the answer as community wiki for the benefit of the community that might encounter this use case in the future.
Feel free to edit this answer for additional information.
I have a PBI desktop dashboard I've created to pull machine data from a local SQL server. I'm using a relative date time filter on one of the pages to drill down data for live feed, however anything under 5 hours of the relative time, the data goes blank.
I use 4 log tables for the raw data, each having their own time stamp for each instance. Each are related using a ID table with other general information contained. In addition, time is related using a calculated table to create a timeframe of all instances:
Relationship Model
DateTable = distinct(union(SUMMARIZE(LogFault,LogFault[Time]),SUMMARIZE(LogGood,LogGood[Time]),SUMMARIZE(LogReject,LogReject[Time]),SUMMARIZE(LogState,LogState[Time])))
5 Hours Relative Time
4 hours relative time
As you can see from the top right of the images, not even the times are pulled to the page. Is there a limitation to PBI on the relative time function? This wouldn't make sense to me if there is a "minutes" option under relative time. Any feedback on this would be appreciated.
For those looking in the future, unfortunately PowerBI desktop, along with service, appears to only like to work in the UTC time zone. So the relative date/time was filtering based on the UTC time zone, not my time zone (EST). In order to resolve this, I had to create a new calculated column next to my distinct time stamps to correct for the time zone. I then used the adjusted time for the relative time filtering, but the charts remained under the original time stamps.
UTC to EST time zone adjust
UTC_AdjustTZ = FORMAT(DateTable[Time]+TIME(4,0,0),"General Date")
Chart Example after adjust
Chart after fix implemented
Probably because your filter on Date Table doesn't reach the destined table. Normally filter moves from one side to many side, then one side to many side in a chain of relationships; but
In your case for example:
Filter goes from Date Table to Log Reject then It can't move to RejectDefinitions because of the filter direction. You have 2 options here:
1) Change the model relationships : Make Log Reject(One side) and RejectDefinitions(Many side) if It is possible.
OR
2) Set the filter direction as Both in the model.
You need to do this for all the remaining log tables(LogFault-FaultDefinitions,Logstate-StateDefinitions)
I hope It solves your problem. Please check that your model is not ambiguous after making those changes.
I have 2 columns one containing order place time and another containing order delivered time and I have created a custom column where we calculate the delivery time in (dd:hh:mm:ss) format now I wanted to calculate average delivery time (i.e average of delivery time in (dd:hh:mm:ss) format).
can any one please suggest me any dax code for the same.
EDIT: I added additional information and context in a new post here.
I am trying to figure out what life after pandemic looks like for my firm.
We are currently operating at about 20% capacity for shipping orders. It is expected that those orders will return after the pandemic subsides.
I have a measure that calculates 80% of our backlogged orders. How to add it to the remaining dates of the year in my model? I assume I need to somehow strip the date context and then parse the total out amongst the remaining days...but I don't have the slightest idea how to begin doing that.
I'd like my total to be spread evenly over the next 4 months. Spearing it evenly by an upper limit works, too.
Any ideas?
Working with unemployment data, and I want to count the number of weeks that a client has received benefits during the last 52 weeks.
I have a dataset with many tables which are all connected via the client number in a fact table. In the fact table the client number occurs for every week of the year in one column, another column shows the type of benefits (if any) received in the current week. So I have a long format.
The other two relevant tables contains
A: List of client number appearing only once per client, but no information about benefits received or week number.
B: List of benefits
Tried counting the rows in my fact table while filtering so as only to do so for the specific benefit I am interested in, but I can't make it work so far.
Weeks on benefit A =
CALCULATE(DISTINCTCOUNT(FactDream[DimDreamBorger]);
DimDreamYdelse[dagpenge] = 1)
sample data
I just need a table showing how many weeks the individual client has received a specific benefit, but I keep getting a total sum of the weeks that all the clients have received the benefit.
Best would be if you add the following measure:
=CountRows(YourTable)
Put this measure into a chart visual. Then add a filter visual and put your Benefit column into the filter visual.
This way you can filter after your benefits, which you are interested, and see the result in the chart.