Spring is really gorgeous here, and very wedding-worthy, although you might get a sprinkling of rain.
I've never planned a wedding here, but you could check out this site for possible locations:
http://weddingswest.com/Victoria/Ceremony_Locations/
If you want old buildings, St. Ann's Academy is beautiful. Do a Google Image search and check out this site:
http://www.stannsacademy.com/Weddings.aspx
I've been in the Swans Hotel wedding suite (top floor loft full of original art, plus rooftop deck) and it's great. You could definitely work out a brunch there, plus it's in Old Town, which is full of old buildings (some lovely, some falling apart but still kinda cool).
Craigdarroch Castle is another possibility. But really, there are so many, and it all depends on your own taste. You should definitely pay a visit and check out a few places in person.
Good luck!