Sourcing new data files automatically from a folder on the cloud

[pete.langlois]

I’m working with a client who will be getting weekly data files in the exact same structure with new records for the past week, and we’d like to create a way in which I do the initial setup of the data model, after which we can simply add new data files to the same folder (same data structure as the previous fact tables) so the new records from each dataset are added to the existing master fact table sitting in the pbix data model.

Is it possible to set up a data sourcing operation in Power BI in which I can drop new CSV files (with new records for a fact table) into a shared folder on the cloud (onedrive, google drive, etc.) and have Power BI automatically source that file, pick up the new records from those new files, and append them to existing fact table already set up in the power bi datamodel? I’m sure this can be done with some additional coding, but I’m hoping there’s an out of the box method someone can share with me.

I understand this can be done in just a few minutes by a power bi developer using power query/append, OR by copy/pasting the new records to the same source file, however I’m hoping there’s an easier way.

If there are other ways to streamline this process of adding regular new data without manually sourcing the new file and running an append, please let me know!

@BrianJ any ideas?

[EnterpriseDNA]

Thanks for posting your question @pete.langlois

Please be reminded that while we encourage everyone’s participation in the forum, we don’t allow single calling out of a member to answer a question as this may discourage other users to help your inquiry.

This is included in the FORUM GUIDE- Asking Questions On The Enterprise DNA Support Forum

Kindly revise your inquiry above. Thank you.

[JasonCockington]

@pete.langlois
File combine is something Power BI does during acquisition. This means it is a Power Query issue.

If you want to use a cloud based folder system, then the easiest option is to use SharePoint, as you can simply connect to a SharePoint folder, and then combine the files.

Is there a reason you are not considering this approach?

[praveer82]

Yes you can do that even if you have data on one drive for business in a folder. Just that reach to the location where your folders are stored and copy the link as mentioned in the pic attached. Ensure to not include anything after underscore …even not underscore.

After that go the power query — get data — sharepoint folder and paste the link you have copied from one

drive url. and after entering credentials you can combine in power query… hope it is helpfull.
Trick is, we can access one drive folder using sharepoint folder connector.
regards

[David]

@pete.langlois

I believe that Ruth (Curbal) has something that would fit what you are after.

A cleaner way to import files from a SharePoint folder with a Power Query function - YouTube

Also Melissa has something with regard to bringing in multiple tabs from an Excel sheet
Append A Vast Number Of Excel Sheets In Power BI - YouTube

Hope this is what you are after.

Regards,

DavidH