Should clients avoid using guards that lost the Guard flag? [current behavior is yes they do]

Nick and I both thought that at least in the past, Tor clients would stop using a relay as their guard, if it loses the Guard flag.

But it looks like the code doesn't do that -- once a relay is your guard, you'll use it in the guard position regardless of whether it has the Guard flag at this moment or not.

This is actually a tricky design decision. In favor of avoiding guards that don't have the guard flag:

• If they get really slow, we can instruct clients to abandon them.
• If a relay gets the guard flag for only a short period of time, it will have only a small number of (dedicated) users using it for the next months.

In favor of using non-Guard guards anyway:

• An attacker can't push you away from your guard by hurting its performance in the eyes of the directory authorities.
• You won't rotate guards as many times.

That "can't push you away" one looks big. What other aspects should we be considering here?