I'm also a grad student, and I would tend to caution against this - although as others have said, you should of course listen to real estate experts as well!
Even though math might be a field with less field work than others, I've still known math grad students who've ended up relocating elsewhere for a semester or year (i.e. if their faculty adviser moved elsewhere for that period or to complete a fellowship), and I've known many students of many disciplines, including math, who end up spending their summers away doing various sorts of summer courses/workshops/etc. at other institutions. It's really nice to be able to either sublet your room in a current rental situation, or to put your things in storage and not worry about rent. While a homeowner could theoretically rent out their house for a summer or a semester, being a landlord is NOT easy and could be particularly tricky if you are not in the area. Even if you have a generous stipend, having to cover a mortgage + rent elsewhere seems like it would be a stretch for anyone. Generally, you're just more flexible, leaving you the opportunity to take advantage of whatever comes up, as well as leaving you an easier exit option if things don't work out with grad school (everyone says this will never happen to them, but it's more common than schools want to tell you!).