Good sources of Data for Modelling


#1

Hi,

I’m not sure exactly in which forum to ask this question but one of the biggest challenges I’m having is finding good sources of data preferably free. For example Bureau of Meteorology data for adding weather data to analysis.

I’d also be interested in ways to connect my own data to power bi such as from strava or apple health kit.

Any help in these areas would be great.


#2

You have access to a lot of demo data here - http://portal.enterprisedna.co/p/demo-data-for-practice

But I presume you’re looking for something different?

Is there something more specific you might be looking for, as it’s quite open ended?


#3

Hi Sam,

I’m looking for a bunch of options at the moment, mainly as I’m spending the majority of my time trying to get access to data rather than working with it.

I’ve had a lot of people ask me to bring weather data into my analysis (I do a lot with councils and their parks and gardens staff doing analysis of gps data). It would be great to do some analysis using both data sets combined. Seeing how weather affects mowing rates etc.


#4

Tough one to advise on. There’s so much out there, some easy and some difficult to connect to.

I might have to just revert to our friend Google on this one.


#5

David ,

Weather channel data used to be free but now that it is owned by IBM it is not. If you can find an API service, I can help you with the code to access it, even if it requires OAuth 2.0 authentication.

Here is a quick dataset solution:
https://home.openweathermap.org/history_bulks/new

They charge a flat fee of $10 no matter how much data you pull for one location.


#6

Hi Tiran,

I think I have found a weather data source from the Australian Bureau of Meteorology (BOM). I’ve been chatting to someone on the phone who was very helpful with giving me all the information except for how to bring the data into Power BI. Here’s a copy of the main information they sent through. They also sent me through a temporary password as well that we could use for the live data. If you send me your email I can send you the password as it updates every Tuesday of the next month.

I’ve kind of brought in the XML file but to be honest I don’t think I’ve done it correctly, just kept hitting the expand button on the columns. If someone could lay out a great process to bring this in that would be awesome.

From the BOM

The data I suggest you use is called our Australian Digital Forecast Database (ADFD). This provides forecasts for the next seven days and is used in all the Bureau’s standard forecast products.

ADFD data (a subset) is visualised in MetEye: http://www.bom.gov.au/australia/meteye/

There is also a location dashboard where you can get tabular details for any location: http://www.bom.gov.au/places/vic/northcote/forecast/detailed/

As discussed, this is gridded data in NetCDF format. Samples are here: ftp://ftp.bom.gov.au/anon/sample/catalogue/ADFD/

Info on parameters available, filenames and other technical details are in the user guide: http://reg.bom.gov.au/catalogue/adfdUserGuide.pdf

I provided a link to a sample file for our live weather station data before, but here it is again for completeness: ftp://ftp.bom.gov.au/anon/sample/catalogue/Observations/IDN60920.xml

As you have geospatial capability you may wish to have a look at our GIS2Web service: http://reg.bom.gov.au/catalogue/GIS2Web.pdf. This is intended for ‘situational awareness’ so doesn’t contain forecast information, but has a good range of current conditions layers, including all major Bureau warning types.


#7

Hi Davd
I normally use below link. Just searched for weather data and so much data, from CSV, Text Files, SQL files, Web Pages etc. And its free. just create the account
https://www.kaggle.com/datasets

Hope this helps.

Shrik