What I want to do is every month or so to get some data from a government site to get some data. At the moment we are doing it by hand copying values from the web site into an Excel. I want to see if I can do this with Power BI.
But this time it’s different, the information from the government site gets replaced, but I want to keep the past values, so when I refresh I get the new value and the past values are not lost.
Not sure it can be done with Power BI ?
Any ideas if it’s possible to refreah without losing past values ?
Here is my Power BI with the web page I’m trying to do.
Assuming you don’t have Pro or Premium or PPU, the Extract the data to excel every time you refresh it. Add the excel data and Online data by appending the tables.
If you have Pro or Premium or PPU then you can use incremental refresh to do that. Or can use dataflow also.
Yes, I have Pro but I have never did this type of refresh. So, I guess it needs to be on the cloud and then get refreshed when I need it. This is not done on Desktop only on the cloud ? Correct ?
Thinking of stopping my Pro because we are have a On-premises License. Can this be done with a On-premises Power BI Report Server ?
If you want to stay current with Power BI feature updates, I would definitely recommend holding on your Pro subscription. There is often a lag time of months from the time a new capability is introduced to the time it shows up in On Prem.
Also, On Prem has a generally more limited feature set (e.g. no R visual integration, more limited on GIS mapping, etc.).
Yes this is only a cloud solution. For desktop you can try the option 1 which is little bit hard to maintain. And I believe it’s not supported as of now based on document, please see below screen shot based on link which I provided you earlier. @BrianJ has given right direction about On-premises vs Pro.
Hi @ericet, did the response provided by the users and experts help you solve your query? If not, how far did you get and what kind of help you need further? If yes, kindly mark as solution the answer that solved your query.
I hope that you are having a great experience using the Support Forum so far. Kindly take time to answer the Enterprise DNA Forum User Experience Survey, we hope you’ll give your insights on how we can further improve the Support forum. Thanks!
Hi @erict. I gave your issue a try, but quickly swam out of my depth in Power Query, so I don’t have a solution for you, but I have encountered this situation frequently before in my historical SQL database work. There, logic like the following was used (using a “Sales” example):
One-time Setup:
create empty table for “New Sales” records
create empty table for “Historical Sales” records
Incremental Refresh:
move rows from “New Sales” table to “Historical Sales” table (“New Sales” table is now empty)
populate “New Sales” table from source
append “New Sales” and “Historical Sales” tables as “Sales” table (use “Sales” for display/calculations)
I’m guessing moving rows/records from one table to another is available in M, but my M knowledge/experience is mostly from the UI and is not nearly as good as my DAX knowledge, so I’ll leave to other forum members to comment further.
Hi @ericet, we’ve noticed that no response has been received from you since the 6th of June. We just want to check if you still need further help with this post? In case there won’t be any activity on it in the next few days, we’ll be tagging this post as Solved.
All is well for me thank you. Figured out that you cannot do incremental on a web import. So no incremental for me.
I was thinking of making a Power BI that will import from the web and then export just the needed data which I will then add to another Excel (Excel incremental ) and load that data in the final Power BI Report.
Not ideal, but importing data from a government web page that changes a couple of times a month and has no place to keep on the government web site a history of the data they published easily accessible. Right now we are manually cut and paste the data from the web page to Excel. So web importing is better then what we are doing now.
So, that’s what I’m figuring out at the moment.
So things I learned;
Incremental is not supported on a web source
You need Power BI Pro to do incremental because the real part of incremental data load happens on the Power BI Online.
Haven’t tried myself but this blog post from Chris Webb talks about something similar using Incremental refresh.
Also, he is using a Web Source but using Web.Contents instead of Web.BrowserContents that I think you can use also as you seems to be loading tables only in your Power BI file.
Do check this out if it is any helpful else I think your solution should be good workaround.
Hi @ericet, did the response provided by the users and experts help you solve your query? If not, how far did you get and what kind of help you need further? If yes, kindly mark as solution the answer that solved your query.
I hope that you are having a great experience using the Support Forum so far. Kindly take time to answer the Enterprise DNA Forum User Experience Survey, we hope you’ll give your insights on how we can further improve the Support forum. Thanks!
) and load that data in the final Power BI Report.