I would think it's the hub/bearing assembly. Not quite sure what you meant by lateral slack in the hub with the wheel off but there shouldn't be any slop in the wheel and if there is the only thing I can think of that could cause that in the rear is the hub/bearing.
It's much more fun in the front, I have some slop in one of my fronts and I had to start from the hub and work my way in to the outer tie rod end to determine that it is in fact the inner tie rod end.