I'd like to see this as well.
To me...it adds a nice touch of realism. And an incentive to try even harder to improve. It would be neat if you were on a bad losing skid, and your home crowd was thinning out with each loss. It would make things a little more satisfying once you started to turn things around, and your home crowd got bigger and more supportive.