A lot of people care because of bragging rights (e.g. UNC behind Dook in polls). But it does matter closer to March for seeding etc. And the AP poll is better indicator of team strength than ESPN/Coach's poll!
Idk if it really matters now, but if you think about it....the higher ranked you are, the better seed you get in the tourney....if you're ranked 16th before the tourney begins, you'll get a 4 seed....etc