Totally read a A Moveable Feast... I lived right across the street from where he wrote it on "Rue Descartes."
One thing you absolutely have to do is check out the neighborhood behind the Pantheon. Start at the Pantheon, and find St. Etienne. Its the most unbelievably beautiful church, and try to be there for an organ improvisation (always on Sunday morning at mass). The two organists there are equally amazing. It sounds like thunder.
Then go and take a right on Descartes, to Place Contrescarpe, which is a cute little square with Cafes and restaurants, then continue down Mouffetard, where there are fun shops (food, drink...). If you are feeling energetic, go back up past contrescarpe, and down descartes, but stay on descartes all the way down to maubert area on St. germain. from there you can explore!
you gotta do this on a weekend, when the Parisians are out and about. Plus you will skip the touristy things which probably suck on weekend days.
good luck.