I can see why you ran into trouble trying to do this without the use of the filter pane. The humble filter pane definitely does not get enough respect. Check out the post below as proof -
using the filter pane - 1 line of DAX
not using the filter pane – 59 lines of complex DAX