User Power query to process multiple excel files and each has multiple worksheets

Every fiscal period an income statement is generated and saved as a XLSX in a sharepoint folder in our organization. Each statement has 140 worksheets representing 140 reporting units in the company. Attached are two sample statements for 10/2022 and 11/2022, but each has only 3 sample worksheets.
PL-Nov2022.xlsx (103.6 KB)
PL-Oct2022.xlsx (108.4 KB)

Every worksheet in all files has the same column structure; it contains two types of P&L: actual and budget. For actual, it has rolling 3 years plus one month of data; for budget, it has the current and the previous years of data. For example, in PL-Nov2022.xlsx, you can find the “actual” numbers from 11/2019 to 11/2022 and the “budget” numbers from 1/2021 to 12/2022.

Here is the requirement I need help with:

  • Every time a new income statement is generated, the actual and budget for a fiscal period of the new file will replace the previous version if any. For example, after PL-Nov2022 is generated, the numbers of every fiscal period from PL-Oct2022 should be deleted except the month of Oct 2022.

My another question is …
How to create a M function to reshape one worksheet’s data at a time? The final look should be like the screenshot below. Currently I append 140 worksheet at once and apply unpivot, fill up/down, etc functions. However, it’s taking way too long to process.

Thank you for your time and help.

Hi Teresa,

I do not work with P&L therefore my assumptions might be wrong.
=> still my approach should work for you

  1. Every month you get a new file => get the latest file from the folder
  2. The only new data is in column 38 of each sheet => Add only this data to your database!
  3. All sheets have the same structure => Use a custom function to get the data from each sheet

For the 3 sample sheets it looks like this => Seems to reflect your screenshot.

I took out calculated rows like margin or total. They are the result of calculations, so they do not belong into your database. As I do not work with P&L I might not have taken all necessary rows respectively have included some not necessary rows.
=> That can be adapted in the custom function!

(table)=>
let
    //Source = LatestFile,
    Data = Table.FirstN(table,124),
    #"Removed Other Columns" = Table.SelectColumns(Data,{"Column1", "Column38"}),
    SheetInfo = Text.Replace(Text.AfterDelimiter(Text.Combine(List.FirstN(List.Skip(#"Removed Other Columns"[Column1]),3) & List.LastN(#"Removed Other Columns"[Column1],1) & List.Range(#"Removed Other Columns"[Column38],5,2), "|"),","),"Generated date: ",""),
    NewData = Table.Range(#"Removed Other Columns",8,91),
    #"Renamed Columns" = Table.RenameColumns(NewData,{{"Column1", "Line Item"}, {"Column38", "Line Item Value"}}),
    #"Filtered null" = Table.SelectRows(#"Renamed Columns", each ([Line Item Value] <> null)),
    #"Replaced Value" = Table.ReplaceValue(#"Filtered null","Net Revenue","Total Net Revenue",Replacer.ReplaceText,{"Line Item"}),
    #"Added Category" = Table.AddColumn(#"Replaced Value", "Category", each if Text.StartsWith([Line Item], "Total ") then Text.AfterDelimiter([Line Item], "Total ") else null),
    #"Filled Up" = Table.FillUp(#"Added Category",{"Category"}),
    #"Filtered Line Items" = Table.SelectRows(#"Filled Up", each not Text.Contains([Line Item], "Margin") and not Text.StartsWith([Line Item], "Total")),
    #"Added Custom" = Table.AddColumn(#"Filtered Line Items", "Custom", each SheetInfo),
    #"Split Column by Delimiter" = Table.SplitColumn(#"Added Custom", "Custom", Splitter.SplitTextByDelimiter("|", QuoteStyle.Csv), {"Fiscal Period End Date", "Currency", "Tab Name", "Statement Generated Date", "Fiscal Period", "Type"}),
    #"Reordered Columns" = Table.ReorderColumns(#"Split Column by Delimiter",{"Type", "Currency", "Fiscal Period End Date", "Statement Generated Date", "Fiscal Period", "Tab Name", "Line Item", "Line Item Value", "Category"}),
    #"Changed Type" = Table.TransformColumnTypes(#"Reordered Columns",{{"Fiscal Period End Date", type date}, {"Fiscal Period", type date}}),
    #"Changed Type with Locale" = Table.TransformColumnTypes(#"Changed Type", {{"Statement Generated Date", type date}}, "en-US")
in
    #"Changed Type with Locale"

You need to adapt the folder path and potentially can take out the first filtering step:

let
    Source = Folder.Files("C:\YourFolderPath"),
    #"Filtered PL-" = Table.SelectRows(Source, each Text.StartsWith([Name], "PL-")),
    #"Sorted Rows" = Table.Sort(#"Filtered PL-",{{"Date created", Order.Descending}}),
    LatestFile = #"Sorted Rows"{0}[Content],
    #"Imported Excel Workbook" = Excel.Workbook(LatestFile)[[Name],[Data]],
    #"Added Custom" = Table.AddColumn(#"Imported Excel Workbook", "NewData", each fxGetData([Data]))[[NewData]],
    #"Expanded NewData" = Table.ExpandTableColumn(#"Added Custom", "NewData", {"Type", "Currency", "Fiscal Period End Date", "Statement Generated Date", "Fiscal Period", "Tab Name", "Line Item", "Line Item Value", "Category"}, {"Type", "Currency", "Fiscal Period End Date", "Statement Generated Date", "Fiscal Period", "Tab Name", "Line Item", "Line Item Value", "Category"})
in
    #"Expanded NewData"

Regards,
Matthias

3 Likes

@Matthias Thank you for the solution and your expertise within 24 hours. Happy New Year!

1 Like

Thank you for the suggested solution.

However, just like what I described in my question, the result should include both actual and budget

Every time a new income statement is generated, the actual and budget for a fiscal period of the new file will replace the previous version if any. For example, after PL-Nov2022 is generated, the numbers of every fiscal period from PL-Oct2022 should be deleted except the month of Oct 2022.

No problem, but the remaining actual months plus budget don’t change, correct?
You complained about performance.
=> Bring that data to a separate (and if possible quicker) source and then combine the old data with new data.

1 Like

Hello @teresating886

Did the responses above help solve your query?

If not, can you let us know where you’re stuck and what additional assistance you need?

If it did, please mark the answer that helped you as the SOLUTION.

Thank you

Hello @teresating886,

Due to the length by which this post has been active, a response on this post has been tagged as “Solution”.

Thanks to the contributors of this post.

For further questions related to this post, please make a new thread. Please feel free to reopen this thread if anyone would like to answer the pending inquiry above.

Thank you