Power Query Workout 13 - Find Correct Department

Difficulty Rating:

Introduction:

You have one table with pay date and employee name. And you need to know the dept the employees worked in depending on the pay date.
There is a second table which has the employee name and an effective date, home dept and worked in dept.
You need a solution which retrieves in table 2 the worked in dept depending on the effective date and employee name.
The generally valid dept information is in the lines with blank effective dates in table 2, the exceptions are stated in the lines with effective dates.

Your task:
Bring the information from the two tables together, showing the actual worked in dept for each pay date.
(NOTE: For more advanced users, attempt to minimize the number of manual changes using relevant M code functions)

13 Find Correct Department.xlsx (20.0 KB)

This workout was posted on Monday June 19, 2023, and the author’s solution will be posted on Sunday June 25, 2023.

5 Likes

Adjunto mi query…

Summary

let
Source = Excel.CurrentWorkbook(){[Name=“Table1”]}[Content],
Type = Table.TransformColumnTypes(Source,{{“Pay Date”, type date}}),
AddCol = Table.AddColumn(Type, “Custom”, (x)=>
let
a = Table.SelectRows(Table2, each (x[Pay Date] = [Effective Date] and [Employee] = x[Employee]) or ([Effective Date] = null and [Employee] = x[Employee])),
b = if Table.RowCount(a) = 2 then Table.SelectRows(a, each [Effective Date] <> null) else a
in b),
Expand = Table.ExpandTableColumn(AddCol, “Custom”, {“Effective Date”, “Home Dep”, “Worked in Dep”}),
Sol = Table.TransformColumnTypes(Expand,{{“Effective Date”, type date}})
in
Sol

2 Likes

@Matthias,

Thanks - another really fun, well-designed problem. It initially looks like a very simple join, but there ends up being a little more to it… :wink:

Click for M Code Solution

let
   Table1 = Table.TransformColumnTypes( Excel.CurrentWorkbook(){[Name="Table1"]}[Content], {"Pay Date", type date} ),
   Table2 =  Table.TransformColumnTypes( Excel.CurrentWorkbook(){[Name="Table2"]}[Content], {"Effective Date", type date}),
   Table2Join = Table.RenameColumns( Table.FillDown( Table.DuplicateColumn( Table2, "Effective Date", "EffDateJoin"), {"EffDateJoin"}), {{"Employee", "Emp"}, {"Worked in Dep", "WorkDep"}}),
    GroupMaxEffDate = Table.Group(Table2Join, {"EffDateJoin", "Emp"}, {{"All", each _, type table [Effective Date=nullable date, Emp=text, Home Dep=number, WorkDep=number, EffDateJoin=nullable date]}, {"MaxEffDate", each List.Max([Effective Date]), type nullable date}}),
    Expand = Table.SelectRows( Table.ExpandTableColumn(GroupMaxEffDate, "All", {"Effective Date", "Home Dep", "WorkDep"}, {"Effective Date", "Home Dep", "WorkDep"}), each [#"Effective Date"] = [MaxEffDate]),
   LeftOuterJoin = Table.RemoveColumns( Table.Join(Table1, {"Pay Date", "Employee"}, Expand, {"EffDateJoin", "Emp"}, JoinKind.LeftOuter), {"Emp", "EffDateJoin"}),
    SortNFillHomeDep = Table.FillDown( Table.Sort(LeftOuterJoin,{{"Employee", Order.Ascending}, {"Pay Date", Order.Ascending}}), {"Home Dep"}),
    AddWorkedInDep = Table.RemoveColumns( Table.AddColumn(SortNFillHomeDep, "Worked in Dep", each if [WorkDep] = null then [Home Dep] else [WorkDep]), {"WorkDep", "MaxEffDate"}),
    Sort = Table.Sort(AddWorkedInDep,{{"Pay Date", Order.Ascending}, {"Employee", Order.Ascending}})
in
    Sort

  • Brian
1 Like
Summary
let
    Fonte1 = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
    Fonte2 = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
    mesc1 = Table.NestedJoin(Fonte1, {"Pay Date", "Employee"}, Fonte2, {"Effective Date", "Employee"}, "Table2", JoinKind.LeftOuter),
    mesc2 = Table.NestedJoin(mesc1, {"Employee"}, Table2, {"Employee"}, "Table2.1", JoinKind.LeftOuter),
    add = Table.AddColumn(mesc2, "Personalizar", each if Table.RowCount([Table2]) > 0 then [Table2] else 
Table.SelectRows([Table2.1], each [Effective Date] = null))
[[Pay Date],[Employee],[Personalizar]],
    res = Table.ExpandTableColumn(add, "Personalizar",{"Effective Date", "Home Dep", "Worked in Dep"})
in
    res
2 Likes

Join, filter, expand.

M Code
let
  Source = Table.NestedJoin(
    Excel.CurrentWorkbook(){[Name="Table1"]}[Content], {"Employee"}, 
    Excel.CurrentWorkbook(){[Name="Table2"]}[Content], {"Employee"}, 
    "Table2", 
    JoinKind.LeftOuter
  ), 
  #"Replaced Value" = Table.ReplaceValue(
    Source, 
    each [Table2], 
    each 
      if List.Contains([Table2][Effective Date], [Pay Date])
      then Table.SelectRows([Table2], (row) => row[Effective Date] = [Pay Date])
      else Table.SelectRows([Table2], (row) => row[Effective Date] = null), 
    Replacer.ReplaceValue, 
    {"Table2"}
  ), 
  #"Expanded Table" = Table.ExpandTableColumn(
    #"Replaced Value", 
    "Table2", 
    {"Effective Date", "Home Dep", "Worked in Dep"}, 
    {"Effective Date", "Home Dep", "Worked in Dep"}
  )
in
    #"Expanded Table"
2 Likes
Summary
let
    T2 = Table.Buffer(Table2),
    Filter = Table.AddColumn(Table1, "T", each let 
        A = Table.SelectRows(T2,(t)=> t[Employee]=[Employee] and (t[Effective Date]<= [Pay Date] or t[Effective Date]=null)) in 
        try A{[Effective Date= [Pay Date]]} otherwise Table.Last(A) ),
    Expand = Table.ExpandRecordColumn(Filter, "T", List.RemoveMatchingItems(Table.ColumnNames(T2),{"Employee"}))
in
    Expand
3 Likes

Hello, nice challenge!
Here is my solution…
Kind regards
Valeria

Summary

let
Source = #“13 Find Correct Department”,

    Table1_Table = Source{[Item="Table1",Kind="Table"]}[Data],

Emp = Table.TransformColumnTypes(Table1_Table,{{“Pay Date”, type date}, {“Employee”, type text}}),

Dept = Source{[Item=“Table2”,Kind=“Table”]}[Data],

#"Date Home Dept." = Table.AddColumn(Emp,"Effective Date Home Dept", each try (Table.SelectRows(Dept, 

(x)=> x[Effective Date]=[Pay Date] and x[Employee]=[Employee]))
{0}[Effective Date]
otherwise null,
type date
),
#“Home Dept” = Table.AddColumn(#“Date Home Dept.”, “Home Dep”, each Table.SelectRows(Dept,
if [Effective Date Home Dept] <> null then
(x)=> x[Effective Date]=[Pay Date] and x[Employee]=[Employee]
else
(x)=> x[Employee]=[Employee])
{0}[Home Dep],type text),

#"Worked in Dep" = Table.AddColumn(#"Home Dept","Worked in Dep", each 

try
Table.SelectRows(Dept,
(x)=>
x[Effective Date]=[Pay Date] and x[Employee]=[Employee]
)
{0}[Worked in Dep]
otherwise
Table.SelectRows(Dept,
(x)=>
x[Effective Date]=null and x[Employee]=[Employee]
)
{0}[Worked in Dep]
,
type text
)
in
#“Worked in Dep”

1 Like

Hello,
thanks to everyone who participated. And a special thanks to all the regular participants. Happy that you showed up again and again. Hope it helped elevating your skills!

Most (all?) of the workouts come from the real world and I love to demonstrate how to solve seemingly difficult problems with relatively easy steps. But you should always try to find alternatives. E.g. here @Luan has a nice, not too difficult logic wich you should bring in to your repertoire.

@AlexisOlson is right that one way to deal with this workkout is about joining, filtering and expanding. My beginner orientated approach brings the filtering behind the expansion.

Let’s start off in Table1 (PayDate and Employee) with selecting on the Home tab the Merge Queries dropdown and then Merg Queries as New:


Choose Table2 and for both queries select Effective date AND Employee:

That will create a new query with a new column Table2 at the end:
image
( Of course you could have merged also in Table1, but I think it is clearer to work in a separate query.)

This new column Table2 we expand, just taking the Effective Date:

Now we merge again, this time we take Merge Queries:


Again choose Table2 and this time select for both queries ONLY Employee:

We’ll get again a new column Table2 and we expand now all columns except Employee:


For Effective Date you will automatically get Effective Date**.1**

Click on column Effective Date and select the top right triangle. Select any of the values:
image

Change in the formula bar that value, with [Effective Date.1]:
image

We don’t need Effective Date.1 any more, so we click on it and press delete.

We are basically ready, but you might want to sort Pay Date and Emplyee in ascending:
image
image

Done!

let
    Source = Table.NestedJoin(Table1, {"Pay Date", "Employee"}, Table2, {"Effective Date", "Employee"}, "Table2", JoinKind.LeftOuter),
    #"Expanded Table" = Table.ExpandTableColumn(Source, "Table2", {"Effective Date"}, {"Effective Date"}),
    #"Merged Queries" = Table.NestedJoin(#"Expanded Table", {"Employee"}, Table2, {"Employee"}, "Table2", JoinKind.LeftOuter),
    #"Expanded Table2" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"Effective Date", "Home Dep", "Worked in Dep"}, {"Effective Date.1", "Home Dep", "Worked in Dep"}),
    #"Filtered Rows" = Table.SelectRows(#"Expanded Table2", each ([Effective Date] = [Effective Date.1])),
    #"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Effective Date.1"}),
    #"Sorted Rows" = Table.Sort(#"Removed Columns",{{"Pay Date", Order.Ascending}, {"Employee", Order.Ascending}})
in
    #"Sorted Rows"

Probably you wouldn’t use this approach for really big data sets, but I like that it is really easy and mostly UI driven. That tweaking in the filtering is very useful and something you can apply again and again.

Try it out and make sure to check also the other approaches above! => Hope you learn something!

Big thanks to everyone who has blurred or hidden the details! It allows readers to develop their own ideas without being influenced - and it’s easy enough to check them afterwards.

Enterprise DNA has decided to temporarily pause the workout program. Let’s see what’s going to happen, in any case we have already a nice collection of 13 workouts: Hope you liked it so far and hope you learnt something. Thanks!

4 Likes

Hello Matthias, thanks!
It is sad the program is paused - I really enjoyed it and I have learned a ton from the answers of the other participants. I actually have a PowerBI file with every workout and the solutions I wanted to learn in a separate query group :grin:
I really hope the program will start again after the summer break. In the meantime, THANK YOU for putting together these nice real world examples!!!
Kind regards
Valeria

1 Like

Thanks for the encouraging feedback, Valeria!
Happy that you enjoyed the workouts and “I have learned a ton” proves that you made good use of of the program. Well done, thanks!

1 Like

Cuanto lamento leer esta noticia @Matthias, la verdad que todos los lunes eatabamos a la espectativa del siguiente reto y sus distintas soluciones. Ojala pronto se reinicien nuevamente :pray::pray::pray:!!!

Igual que @valeriabreveglieri, agradezco el tiempo y el ezfuerzo que desinteresadamente le han puesto para presentarnos estos retos…

2 Likes

Gracias Alejandro! Tienes razĂłn: Es realmente lamentable y ojala pronto se reinicien nuevamente.

We hope you come back soon!

2 Likes