Refreshing of only Delta data in Power BI model

Hi,
I have a sales dataset of over 4years and the row counts is in some millions. There is a chance of this count increasing to billions in future. So, when refreshing every time, the model refresh is taking lots of time just to import the data. How can I solve this? Is there anything in Power BI, just to import the recently added/updated data without importing all historic data ? If yes, please let me know how to achieve that.
Thanks in advance.

Thanks,
Anusha

Hi @Avenkataanusha - depending on where your data is stored, you may have the option of using Direct Query. There are some limitations with Direct Query beyond the restrictions on which platform it can be used, but that would be your best start.

Hope this helps
David

1 Like

Hello @Avenkataanusha,

Thank You for posting your query onto the Forum.

Well you can try the “Incremental Refresh” option. I’m providing a link below of an article from the Microsoft based on this topic.

Earlier this option was only available under the “Premium” version but was then made available for the “Pro Users” as well since Feb, 2020.

Hoping you find this useful and meets your requirements that you’ve been looking for. :slightly_smiling_face:

Thanks and Warm Regards,
Harsh

2 Likes

Hi,
Thank you so much for this. I have a question here. Can we set incremental refresh on multiple sources in same data model?

Thanks,
Anusha

Hi David,

Thanks for the reply. With direct query, the report is taking some time to load for each sicer selection, which is a bit worry for the clients. Is this normal in any case?

Thanks,
Anusha

Hello @Avenkataanusha,

Yes, you can set up incremental refresh for multiple data sources. I’m providing a link below of an article from RADACAD. And also providing some of the links of the videos on Incremental Refresh.

Please Note: The article is was published in June, 2019 so at that time this option was available only to Premium Users but now it’s available to Pro Users as well. Apart from this, everything absolutely remains the same. So you can go through this article and set up incremental refresh for multiple data sources.

Hoping you find this useful. :slightly_smiling_face:

Thanks and Warm Regards,
Harsh

Hello @Avenkataanusha,

I’m providing a link below where a member had asked onto the Forum whether they should go for the DirectQuery or DataImport option and we’d a great discussion at length on this topic where some good points were exhanged.

I’m providing a link below of that post.

Hoping you find this useful and helps you to choose the better option in your scenario. :slightly_smiling_face:

Note: Please click onto the link to view the entire post and not on the expand/collapse button.
.
Thanks and Warm Regards,
Harsh

@Avenkataanusha :
I had done quite a bit of reading on DirectQuery Vs. DataImport when was tasked to do the conversion for my client. And also work with MSFT clinic gurus for advice.
Here is the summary to share:-

  1. When your dataset will increase to the size it max out your PowerBI service capacity, you have no choice but to take the largest table you have in your model, to convert to DirectQuery
  2. DirectQuery is always the slowest in perf. To improve the perf in DQ is always a big task. The key to the tuning is to find ways to reduce # of query calls. You scrutinize each line of your DAX code in Dax Studio. (of course network traffic, location of your db…etc also contribute to your report perf)
  3. If report is complex and perf is the key requirement, always look into “Composite model” as your alternative, no need to go with pure DirectQuery.

I hope this is helpful.

Hi @Avenkataanusha, a response on this post has been tagged as “Solution”. If you have a follow question or concern related to this topic, please remove the Solution tag first by clicking the three dots beside Reply and then untick the check box. Also, we’ve recently launched the Enterprise DNA Forum User Experience Survey, please feel free to answer it and give your insights on how we can further improve the Support forum. Thanks!