Power Query Workout 03 - List the Employee's Bonus Points

Difficulty Rating:

Introduction:

Every day the employees of our store can earn bonus points, which do have an influence on their monthly pay. So it is extremely important that you report the information correctly.
Unfortunately the format in which the bonus points are logged per date and category isn’t regular, so you need to transform the data.

Your task:
Build a dynamic solution which is independent from the number of categories and the number of employees in the data.
This is an intermediate - advanced workout => But you can still build a UI driven solution.

(NOTE: For more advanced users, attempt to minimize the number of manual changes using relevant M code functions)


3 List Bonuspoints.xlsx (13.4 KB)

This workout was posted on Monday April 10, 2023, and the author’s solution will be posted on Sunday April 16, 2023.

1 Like

@Matthias ,

Great workout. This sort of terrible data structure is perfect for PQ to clean up.

I am very curious to see how you solve this one solely with the UI, since I relied heavily on M code to test the type of data in a given column.

Here’s my solution:

let
    Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
    AddDate = Table.AddColumn(Source, "Date", each if Value.Is([Column1], DateTime.Type) then [Column1] else null),
    FillDown = Table.FillDown(AddDate,{"Date"}),
    ChangeType = Table.TransformColumnTypes(FillDown,{{"Date", type date}}),
    FilterNulls = Table.SelectRows(ChangeType, each ([Column2] <> null)),
    AddCategory = Table.FillDown( Table.AddColumn(FilterNulls, "Category", each if [Column1] = "Category" then [Column2] else null), {"Category"}),
    FilterNonData = Table.SelectRows(AddCategory, each ([Column1] <> "Category" and [Column1] <> "Employee")),
    UnpivotOther = Table.RemoveColumns( Table.UnpivotOtherColumns(FilterNonData, {"Date", "Category"}, "Attribute", "Value"), "Attribute"),
    AddEmployee = Table.FillDown( Table.AddColumn(UnpivotOther, "Employee", each if Value.Is( [Value], Text.Type ) then [Value] else null), {"Employee"}),
    AddPoints = Table.FillUp( Table.AddColumn(AddEmployee, "Points", each if Value.Is( [Value], Number.Type ) then [Value] else null), {"Points"}),
    Clean = Table.Distinct( Table.RemoveColumns(AddPoints,{"Value"}) )
in
    Clean

3 List Bonuspoints - Brian Julius Solution.xlsx (23.5 KB)

1 Like

@Matthias -

Have to say this one was a bit of a doozy. And I was about to throw in the towel as it was getting late, but stuck with it and look like I got it (and even tested it by adding more data).

@BrianJ - my approach is about 95% using UI - and I had to adjust a little of the M code in a number of the “Replaced Value” steps (mainly because I wanted to use some conditional statements for the Replace Values and the standard UI function doesn’t provide that).

Here’s my solution:

Workout 003 Pic 1

2 Likes

My Submission -

let
Source = Excel.CurrentWorkbook(){[Name=“Table1”]}[Content],
ExpectedOutput = Table.FromRecords(List.Combine(Table.Group(Source, {“Column1”}, {{“All”, (g_1)=> List.Combine(Table.Group(Table.Skip(g_1), {“Column1”}, {{“All”, (g_2)=> List.Transform(List.Split(List.Select(Table.ToColumns(Table.Skip(g_2)), (n)=> List.NonNullCount(n) > 0),2), (t)=> [l = List.Zip({{“Date”, Date.From(Table.FirstValue(g_1))}} & {List.FirstN(Record.ToList(g_2{0}),2)} & t), r = Record.FromList(l{1},l{0})][r])}}, 0, (a,b)=> Number.From(b[Column1] = “Category”))[All])}}, 0, (x,y) => Number.From(y[Column1] is datetime))[All]))
in
ExpectedOutput

2 Likes

Adjunto mi query…

let
Source = Excel.CurrentWorkbook(){[Name=“Table2”]}[Content],
DateCol = Table.AddColumn(Source, “Date”, each if Value.Type([Column1]) = type datetime then [Column1] else null),
CatCol = Table.AddColumn(DateCol, “Category”, each if [Column1] = “Category” then [Column2] else null ),
Fill = Table.FillDown(CatCol,{“Date”, “Category”}),
Remove1 = Table.SelectRows(Fill, each Value.Type([Column1]) <> DateTime.Type and [Column1] <> “Category”),
Encabezados = List.Select(Table.ColumnNames(Remove1), each not Text.Contains(_, “Col”)) & List.Select(List.Distinct(List.Combine(Table.ToRows(Table.SelectColumns(Table.AlternateRows(Remove1, 1, 1, 1), Table.ColumnNames(Source))))), each _<> null),
RemAlt1 = Table.AlternateRows(Remove1, 0, 1, 1),
Unpivot = Table.UnpivotOtherColumns(RemAlt1, {“Date”, “Category”}, “Attribute”, “Value”),
Index = Table.AddIndexColumn(Unpivot, “Index”, 1, 1, Int64.Type),
Alinear = Table.AddColumn(Index, “Custom”, each Unpivot[Value]{[Index]}),
RemAlt2 = Table.AlternateRows(Alinear,1,1,1),
RemCol = Table.FromColumns(Table.ToColumns(Table.RemoveColumns(RemAlt2,{“Attribute”, “Index”})), Encabezados)
in
RemCol]

2 Likes
let
    Fonte = Tabela1,
    rec = Table.AddColumn(Fonte, "Personalizar", each [
    Date = if [Coluna1] is datetime then [Coluna1] else null,
    Category = if [Coluna1] = "Category" then [Coluna2] else null
    ]),
    exp = Table.ExpandRecordColumn(rec, "Personalizar", {"Date", "Category"}),
    pc = Table.FillDown(exp,{"Date", "Category"}),
    fil = Table.SelectRows(pc, each  [Coluna1] is text and [Coluna1] <> "Category"),
    res = Table.AddColumn(fil, "Personalizar", each List.RemoveNulls(List.RemoveLastN( Record.FieldValues(_),2)))[[Date],[Category],[Personalizar]],
    ind = Table.AddIndexColumn(res, "index", 0, 1, Int64.Type),
    tab = Table.AddColumn(ind, "Personalizar.1", each try ind{[index]+1}[Personalizar] otherwise null),
    list = List.Select(tab[index], each Number.IsEven(_)),
    cr = tab,
    filt = Table.SelectRows(cr, each List.Contains(list,[index] ) )[[Date],[Category],[Personalizar.1]],
    ex = Table.ExpandListColumn(filt, "Personalizar.1"),
    re = Table.AddColumn(ex, "Personalizar", each [
    Employee = if [Personalizar.1] is text then [Personalizar.1] else null,
    Points = if [Personalizar.1] is number then [Personalizar.1] else null

    ]),
    expa = Table.ExpandRecordColumn(re, "Personalizar", {"Employee", "Points"}),
    pb = Table.FillDown(expa,{"Employee"}),
    fi = Table.SelectRows(pb, each ([Points] <> null)),
    result = Table.SelectColumns(fi,{"Date", "Category", "Employee", "Points"})
in
    result
3 Likes

@Matthias, thanks for a good workout.

@BrianJ except for line 4, I only used UI

Here is my solution:


image

2 Likes

@KimC,

Nice! @tweinzapfel figured out the UI hack for Value.Is

Change a column to a particular type in the UI (say, date). All other types will error out. Then replace the errors with nulls. Clever…

1 Like

Donn Clark
Apr 11, 2023

Solution to #3

Thanks

2 Likes

I thought you suppose show(blurr) your code how you got that your results?

@Keith you can either hide or blur by clicking on the cog on the top right:
image

Hello, this is the way I did it. I was not sure if we were meant to tidy up the data from the worksheet as well… I just took it as another challenge! To try out something I had found on linkedin using data types (link )

Thanks for the challenge! I am super curious now to go and look at the other solutions :slight_smile:

Code is here:

let
Source = Excel.Workbook(File.Contents(“mypath\Downloads\3 List Bonuspoints.xlsx”), null, true),
#“Employee’s Bonuspoints_Sheet” = Source{[Item=“Employee’s Bonuspoints”,Kind=“Sheet”]}[Data],
#“**** This is the code to extract the table from the Excel Worksheet” = #“Employee’s Bonuspoints_Sheet”,
#“Added Custom” = Table.AddColumn(#“Employee’s Bonuspoints_Sheet”, “Data Type”, each Table.Schema(#table(type table [C1=Value.Type([Column1])],{})){0}[Kind],type text),
#“Reordered Columns” = Table.ReorderColumns(#“Added Custom”,{“Data Type”, “Column1”, “Column2”, “Column3”, “Column4”, “Column5”, “Column6”, “Column7”, “Column8”, “Column9”, “Column10”, “Column11”, “Column12”, “Column13”, “Column14”}),
#“Skip First Rows Dynamically” = Table.Skip(#“Reordered Columns”,List.PositionOf(#“Reordered Columns”[Data Type],“date”)),
OriginalColumnNames=Table.FromList( Table.ColumnNames( #“Skip First Rows Dynamically”)),
#“New Column Copied” = Table.DuplicateColumn(OriginalColumnNames, “Column1”, “New Column Names”),
#“Numbers Only” = Table.TransformColumns(#“New Column Copied”, {{“New Column Names”, each Text.AfterDelimiter(_, “Column”), type text}}),
OldColumnNames = List.Zip({#“Numbers Only”[Column1],#“Numbers Only”[New Column Names]}),
#“Renamed Table” = Table.RenameColumns(#“Skip First Rows Dynamically”,OldColumnNames),
#“Changed Type” = (Table.TransformColumnTypes(Table.Profile( #“Renamed Table”),{{“Column”, Int64.Type}})),
#“Check null columns” = Table.SelectRows(Table.AddColumn(#“Changed Type”, “Subtraction”, each [NullCount] - [Count], type number),each [Subtraction]=0),
#“Min of null column” = Table.Min( #“Check null columns”,each [Column])[Column],
#“**** Here Starts the code to clean up the data”= #“Min of null column”,
#“Removed Other Columns” = Table.RemoveColumns(#“Skip First Rows Dynamically”,List.Range( Table.ColumnNames( #“Skip First Rows Dynamically”), #“Min of null column”)),
#“Added Custom1” = Table.FillDown(Table.AddColumn(#“Removed Other Columns”, “Date”,each if ([Column1]) is date then [Column1] else null, type date),{“Date”}),
#“Filtered Rows” = Table.SelectRows(#“Added Custom1”, each ([Data Type] = “text”)),
#“Removed Columns” = Table.RemoveColumns(#“Filtered Rows”,{“Data Type”}),
#“Added Custom2” = Table.FillDown(Table.AddColumn(#“Removed Columns”, “Category”, each if [Column1]=“Category” then [Column2] else null),{“Category”}),
#“Filtered Rows1” = Table.SelectRows(#“Added Custom2”, each ([Column1] <> “Category” and [Column1] <> “Employee”)),
#“Unpivoted Other Columns” = Table.UnpivotOtherColumns(#“Filtered Rows1”, {“Date”, “Category”}, “Attribute”, “Value”),
#“Removed Columns1” = Table.RemoveColumns(#“Unpivoted Other Columns”,{“Attribute”}),
#“Added Custom3” = Table.FillUp(Table.AddColumn(#“Removed Columns1”, “Points”, each if [Value] is number then [Value]else null),{“Points”}),
Custom1 = Table.SelectRows(#“Added Custom3”,each [Value] is text)
in
Custom1

2 Likes
let
    Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
    Group = Table.Group(Source,"Column1",{"T",each let T = Table.ToRows(_)  
        in  try Table.FromRows(List.Transform(List.Split(List.RemoveNulls(T{2}),2), each {T{0}{1}}&_), {T{0}{0}}& List.FirstN(T{1},2)) 
        otherwise Table.RenameColumns([[Column1]],{"Column1","Date"})},0,(b,e)=> try Number.From(e) otherwise Number.From(e="Category") ),
    Fill = Table.SelectRows(Table.FillDown(Table.Combine(Group[T]),{"Date"}),each [Category]<>null)
in
    Fill
4 Likes

Hello,
thanks to everyone who participated! Everyone who solved this advanced workout is a winner!

As you can see above there are many possibilities with varying degrees of difficulty to solve the workout. I’ll present two options which rely heavily on Table.AddColumns. With the first one I present a neat technique which @valeriabreveglieri used to indentify data types Rick de Groot and with the second I present a useful pattern which allows you to shift a column at ease Imke Feldmann.

Both options start off with two Added Columns: Go to the Add Column tab and chose Custom Column. First time we want to identfy the dates from Column1 and if there is a date we take it over and if not we take null. We can use this for the task:
try Date.From([Column1]) otherwise null
It makes sense to add also type date.

Second time we want to take over the Category from Column2, so if there is “Category” written in Column1 we take Column2 and if not we take null:
if [Column1]=“Category” then [Column2] else null

Next we fill down the values on both Added Customs. Mark both columns and then right click and select Fill > Down

Then we need to select the relevant rows. We don’t want the rows with Emplyoyee or Category and we don’t want empty rows:
each [Column1] <> “Employee” and [Column1] <> “Category” and [Column2] <> null

From here you have several options to deal with the data. Grouping would be a common one, but I think Unpivoting probably offers the easiest way forward. Mark the last 2 columns before going to the Transform tab to select Unpivot Columns > Unpivot Other Columns.
Change “Value” in the formula bar to “Employee”.

Then let’s mark the Attribute column and press Delete to remove it.

So far we had 6 relatively easy steps - only the first one is a bit more avanced.

let
    Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],

    #"Added Date" = Table.AddColumn(Source, "Date", each try Date.From([Column1]) otherwise null, type date),
    #"Added Category" = Table.AddColumn(#"Added Date", "Category", each if [Column1]="Category" then [Column2] else null),
    #"Filled Down" = Table.FillDown(#"Added Category",{"Date", "Category"}),
    #"Filtered Rows" = Table.SelectRows(#"Filled Down", each [Column1] <> "Employee" and [Column1] <> "Category" and [Column2] <> null),
    #"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Filtered Rows", {"Date", "Category"}, "Attribute", "Employee"),
    #"Removed Columns" = Table.RemoveColumns(#"Unpivoted Other Columns",{"Attribute"}),
.....

First Option:
We use another Added Column for the Points. So go to the Add Column tab and chose Custom Column. This time we want to take over all number from the Employee column and if there is a name we take null. If you think about it we could use a variant from the technique in step one for it (try it yourself), but this super short code will do the trick - you need to learn it:
if [Employee] is text then null else [Employee]

Then fill up the values on the new Added Custom. Mark the column, right click and then select Fill > Up

Finally we need to select all rows where Employee is not equal to Points. Select any of the columns and select any of the values and then bring this to the formula bar: each [Employee] <> [Points]

Done! What I like about it is that it contains incredible useful concepts you can apply over and over again and at the same time is should be still understandable even for not so advanced users.

    #"Added Custom" = Table.AddColumn(#"Removed Columns", "Points", each if [Employee] is text then null else [Employee]),
    #"Filled Up" = Table.FillUp(#"Added Custom",{"Points"}),
    #"Filtered Rows1" = Table.SelectRows(#"Filled Up", each [Employee] <> [Points])
in
    #"Filtered Rows1"

Second option:
You need to learn this Shift pattern from Imke Feldmann, so let’s use this task as an excuse. I marked the prevStep the to be shifted ColumnName and the resulting shifted column in italics:
Table.FromColumns(
Table.ToColumns(#“Removed Columns”) & { List.Skip(#“Removed Columns”[Employee]) },
Table.ColumnNames(#“Removed Columns”) & {“Points”} )

Then you just need to get rid of every second row (Home tab > Remove Rows > Remove Alternate Rows) or you selct rows with the thechnique seen above (each [Employee] is text) or …
[100% clicking fun: Alternatively right click on the Points column and Change Type to Whole Number. Then click on the Home tab > Remove Rows > Remove Errors.]

Done! The shift pattern is a bit more complex, but it is incredible useful, so worth learning it.

    Shift = Table.FromColumns(
        Table.ToColumns(#"Removed Columns") & { List.Skip(#"Removed Columns"[Employee]) },
        Table.ColumnNames(#"Removed Columns") & {"Points"} ),
    #"Filtered Rows1" = Table.SelectRows(Shift, each ([Employee] is text))
in
    #"Filtered Rows1"

Try both options, but make sure to compare them also to the other solutions, following completely different approaches! => Hope you learn something!

Big thanks to everyone who has blured or hidden the details! I have taken this now out so that it’s easier to compare solutions.

1 Like

@Matthias ,

Great problem, terrific solution breakdown and great work done by everyone in the community in providing different and very creative solutions - exactly how I’d hoped these workouts would go when we started them. Thanks to all!

  • Brian
1 Like

let
Source = Excel.CurrentWorkbook(){[Name=“Table1”]}[Content],
#“Duplicated Column” = Table.DuplicateColumn(Source, “Column1”, “Date”),
#“Changed Type” = Table.TransformColumnTypes(#“Duplicated Column”,{{“Date”, type datetime}}),
#“Replaced Errors” = Table.ReplaceErrorValues(#“Changed Type”, {{“Date”, null}}),
#“Filled Down” = Table.FillDown(#“Replaced Errors”,{“Date”}),
#“Added Custom” = Table.AddColumn(#“Filled Down”, “Custom”, each if[Column1]=[Date] then null else[Column1]),
#“Filtered Rows” = Table.SelectRows(#“Added Custom”, each ([Custom] <> null)),
#“Removed Columns” = Table.RemoveColumns(#“Filtered Rows”,{“Custom”}),
#“Filtered Rows1” = Table.SelectRows(#“Removed Columns”, each ([Column2] <> “Points”)),
#“Added Custom1” = Table.AddColumn(#“Filtered Rows1”, “Category”, each if [Column1] = “Category” then [Column2] else null),
#“Filled Down1” = Table.FillDown(#“Added Custom1”,{“Category”}),
#“Filtered Rows2” = Table.SelectRows(#“Filled Down1”, each ([Column1] <> “Category”)),
#“Unpivoted Other Columns” = Table.UnpivotOtherColumns(#“Filtered Rows2”, {“Category”, “Date”}, “Attribute”, “Employee”),
#“Added Custom2” = Table.AddColumn(#“Unpivoted Other Columns”, “Points”, each if[Employee] > 0 then [Employee] else null),
#“Replaced Errors1” = Table.ReplaceErrorValues(#“Added Custom2”, {{“Points”, null}}),
#“Filled Up” = Table.FillUp(#“Replaced Errors1”,{“Points”}),
#“Added Custom3” = Table.AddColumn(#“Filled Up”, “Custom”, each if [Employee] = [Points] then 1 else 0),
#“Filtered Rows3” = Table.SelectRows(#“Added Custom3”, each ([Custom] = 0)),
#“Removed Columns1” = Table.RemoveColumns(#“Filtered Rows3”,{“Attribute”,“Custom”}),
#“Changed Type1” = Table.TransformColumnTypes(#“Removed Columns1”,{{“Date”, type date}})
in
#“Changed Type1”

1 Like

This one wasn’t the easiest one, great workout Matthias! Another option is this solution:

let
    Source = Excel.CurrentWorkbook(){[Name="MyData"]}[Content],
    AddDate = Table.AddColumn(Source, "Date", each if [Column1] is datetime then [Column1] else null, type datetime ),
    AddCat = Table.AddColumn(AddDate, "Category", each if [Column1] = "Category" then [Column2] else null),
    FillDown = Table.FillDown(AddCat,{"Date", "Category"}),
    KeepNumbers = Table.SelectRows(FillDown, each [Column2] is number ),
    Unpivot = Table.UnpivotOtherColumns(KeepNumbers, {"Date", "Category"}, "Attribute", "Value"),
    Repl = Table.ReplaceValue(Unpivot, each [Attribute], each let n =  Number.From( Text.End( [Attribute], 1) ) in if Number.IsEven( n ) then n-1 else n ,Replacer.ReplaceValue,{"Attribute"}),
    Group = Table.Group(Repl, {"Date", "Category", "Attribute"}, {{"Count", each Record.FromList( _[Value], {"Name", "Points"} ), type record}}),
    Expand = Table.ExpandRecordColumn(Group, "Count", {"Name", "Points"}, {"Name", "Points"}),
    Remove = Table.RemoveColumns(Expand,{"Attribute"})
in
    Remove

Hi @Matthias,

Here is my solution to this workout.

I’m working through them.

Thanks
Keith