I'd like to see this as well.
To me...it adds a nice touch of realism.  And an incentive to try even harder to improve.  It would be neat if you were on a bad losing skid, and your home crowd was thinning out with each loss.  It would make things a little more satisfying once you started to turn things around, and your home crowd got bigger and more supportive.