Hello Nadine,
I think the problem is that it never sorts by affiliation. As you mention, the first sorting factor is a taxonomy. But you need to specify the specific factor of the taxonomy vocabulary that you want it to sort. Is it the name of the terms? Is it the creation date? The order of the taxonomy terms?
Drupal by default does not understand the factor if you dont specify it. In order to do that you need to setup a relationship in your view to the taxonomy vocabulary that you want to sort by. You can have a look on a topic where I answered how to sort by order of taxonomy terms. In a similar way you should create a relationship and then add the sorting factor from the taxonomy vocabulary.
Let me know if it is clear. Also if it does not work, please specify the view URL to have a better look.
Konstantinos