A new year is coming and the path where we save our data is changing.
The problem is: the data from each asset is saved in folders with the current year as it's name, so data from 2022 is saved in a folder called 2022, and so on.
I want to make a query that will autamatically indentify what years (folders) we have data from and combine them.
The data is saved in this path:
C:\Users\Projects\3. Assets\Type A\Asset Name\Control\YEAR\Data\Dataset\excel.xlsx
This asset for instance has 3 years of data 2020, 2021 and 2022.
By next week we will already have a 2023 folder with new data, usually a manually add a table.combine to the query, but we have a large number of assets and it can be tricky.
Someone knows a efficient way to automatically identify all the folders named with a year and combine the excel data inside them ?
This is the way i usually do:
Table.Combine ({Sharepoint("...2020/Data/Dataset"),Sharepoint("...2021/Data/Dataset"),Sharepoint("...2022/Data/Dataset")})
Sharepoint is a function that returns folder content from sharepoint.
Best Regards
Can you pull all the directory names at a higher level
similar to
https://exceltown.com/en/tutorials/power-bi/powerbi-com-and-power-bi-desktop/power-bi-data-sources/connect-power-query-whole-sharepoint-folder/
then filter them ?
Related
new to Power BI and just built a dashboard with some finance data with the following columns,
Date|Transaction ID|Transaction Amount|Item Description|Item Key
Every month I receive a new CSV file with data for the previous month. Rather than manually adding the new data to a master file each month, is there a way to simply drop the new CSV file into a folder each month and then refresh the dashboard so it automatically includes the new data (minus the headers)? If possible, I'd also like to add a column which holds the date the new file was loaded, so each new month's file is date stamped each time it's added.
Many thanks
What you can do is use a folder as a source instead of a csv.
That folder should contain all csv files.
When all your files are load you only have to select the following option:
After that you will have all the data from the all csv files in one gigant table.
Unfortunately its not possible to add a date column with the load date.
The only way to do that is that the csv files have the date column.
Hope it helps you.
I'm brand new to Power BI and I'm used to setting up most of my data in SQL Server (for SSRS). I have a data set and I was able to add in a Calendar table with my dates. My goal is to do a Year-over-year comparison. I got the year-over-year part working with the help of a couple of tutorials, but I want to restrict the report output to only data up to the last end of month (otherwise the YoY shows a case differential for dates out into 2021 - not helpful). I need a dynamic filter and all I seem to be able to set are static filters. This filter needs to be on the data itself - nothing a user can or should touch. Any help would be appreciated.
I have files in a folder which have same structure. The only difference between them is that for each day a new file is created which is named as date of the day.
So if file is created on 11th November 2019, its name would be 11092019.xlsx.
I have created a slicer which fetches names of all files present in this folder.
Keeping in mind that file format is same and only difference is in their file name and data values in them. Is there any way so that when I select a value from slicer the respective files data will be displayed in table visual ?
It is not possible to load a file interactively based on the slicer value.
You can achieve this by,
Load all files in the folder.
Combine them into a single table, where you need to add a "File Name" column.
Use slicer to show records coming from the specific file.
We are utilizing an excel monthly report in our power bi project that has added measure columns, and we keep the sheets the fields pull the data from in one folder. When we get each month's updated excel sheet, would we be able delete the old one, add the new report to the folder with the exact same name as the old, and refresh the power bi query to use the new updated data? All the column headers would remain the same, the only thing that would be changing is maybe the amount of rows and the data within them. If we were to keep all the names the same, the only thing changing is the data sheet itself (not the column headers just the data) would the added measure columns remain and work? The measure columns act as column data multipliers and filters, and it would be a pain to make new ones each month.
Thanks
Yes. If the file path and filename and sheet/table name all remain the same, Power BI won't know the difference and you shouldn't have trouble if the columns and headers stay consistent.
Additionally, if you don't want to rename the file or delete/move older files from the folder, you could do a Load from Folder query and sort by date created/modified and grab the top row instead of specifying the filename.
I've been working on a SAS ETL project wherein we first extract data for the last month from teradata warehouse in the beginning of every month and then take it further for processing.
This is done via extraction scripts for each table and then the data is stored into a monthly folder structure (yyyymm). After working this way for several months, we've now begun getting requests to product daily, weekly, etc extracts.
The current data storage folder structure is :
Library/Data/YYYYMM folder in one library.
I have to change the structure(with minimal impact to the current structure) to accommodate different timeframe requests like Daily, weekly, Fortnightly, Quarterly, etc.
I thought of two options : In the current structure (monthly folders), add in
Daily, Weekly and Monthly Folders
Library / YYYYMM / Monthly
Library / YYYYMM / Daily
Library / YYYYMM / Weekly
folders.
Option 2:
Under the Data Library Create folders like
Monthly
Daily
Weekly
fortnightly
Quarterly
Under each of these exists individual folders with the current date/month/quarter.
Can anyone suggest of any other more practical design approaches?
Maybe SAS generation data sets would be an option: http://support.sas.com/documentation/cdl/en/lrcon/62955/HTML/default/viewer.htm#a000934566.htm
Here an extract from SAS:
A generation data set is an archived version of a SAS data set that is stored as part of a generation group. A generation data set is created each time the file is replaced. Each generation data set in a generation group has the same root member name, but each has a different version number. The most recent version of the generation data set is called the base version.