What makes a book valuable is when more people want to buy it than there are copies readily available. It's very difficult to predict. Generally speaking, books with small print runs are going to be more valuable than ones with large ones; but the vast majority of books of any kind, especially in the internet age, turn out to be completely worthless almost as soon as they are bought.
Signed copies are not in themselves valuable; it has to be an author with some cachet, or scarcity. In that case, it can make a big difference. Presentation copies (signed, with a handwritten dedication to another famous person) can be valuable too, but in general you sound like you're not in the market for signed editions, unless you are getting them signed yourself.
The bog-standard "collectible book" is a modern first edition, i.e., a novel written by an author of high stature. First novels are often good, if (but only if) you hit upon a first novel by a nobody in a tiny print run who turns out to be a superstar later. But if you just bought every first as it came out your investment would be wasted, because most of them will be worth $0.01 on Amazon within a year.
The other thing that matters so much is condition, condition, condition. Perfect books hold their value, and maybe appreciate; bent, knocked, or dented ones don't. It's as simple as that.
Genre fiction can be a good area to collect in, especially sci fi. Another area where books often appreciate is photography books, monographs of top photographers. They go out of print, and if the photographer is hot they can then sell for a ton of money. But again, you have to guess who's going to be hot.
In the internet age, all sorts of oddities are turning out to be hugely valuable, while the wide availability of other stuff that had previously been assumed to be fairly reliable has collapsed its prices. For instance, I've tried to sell some books in the past, and the only two I ever made a quick killing on were a rare aquarium book by this Japanese guy, and a collector's guide to Hamilton watches. I was shocked -- they both sold for over $100 within minutes after being listed. Loads of other really good collectible stuff never attracted any interest at all. Supply and demand.
The best place to learn is used bookstores, specifically ones that do a lot of collectible business. Go to Wessel & Lieberman in Pioneer Square, or John Michael Lang in Ballard, and just browse. Browse and browse and browse. Look at the books. Look at the prices. Find similar books where one is $10 and the other is $200, and see if you can tell why. Look on Ebay, and sort the books section by price, and see what goes for a lot of money, and why. Get ahold of bookseller catalogs, too -- same thing. Also, learn the lingo -- learn what words like "rubbed" and "chipped", "octavo" and "12mo" mean. Look on sites like Abebooks and see the difference in price between different editions (or even the same one) of something.
The trick is to find out what's available now for cheap that will be dear tomorrow. The only way to really do that is to learn the business a little. Ask questions; a lot of booksellers love to jabber endlessly about their trade, and most of them are pretty smart.
In addition to knowing books, you have to know your subject. If you're just wandering into B&N trying to guess what's going to make you rich, forget it. If you're buying SF, you have to REALLY KNOW SF, and you have to be able to tell what's good and what isn't. Same with photography or anything else. What are you interested in? Those are the books you should be looking at.
You probably won't find anything you're interested in on this blog, but you should read it, for his insights into what makes books worth more or less, and how that changes over time: http://www.bookride.com/
Go to the library and check out a few book-collecting books, starting with the classic "ABC for Book Collectors" by John Carter and Nicolas Barker and "Book Finds" by Ian Ellis, or whatever else they have. Be prepared for a lot of stuffy, stuck-up guff from some of these books, which sometimes appear to be written by semi-senile old duffers droning on and on about that 1732 Principia Mathematicus by Sir Ethelred Bunnywhip QV RSC Msc.PRSD that they spilled port on during that terrible thunderstorm in 1919, blah de blah de blah, but there's some good ones too. Carter is great.
Whatever you do, don't turn into one of those pricks who goes to yard sales and thrift stores with a scanner and scans everything and leaves it in a big pile afterwards. Everybody hates those guys. Yard sales are a terrible place to look for books, anyways, unless you're looking for 1980s college textbooks or the complete sermons of Billy Graham in two hundred baby-blue volumes. The only time you'll ever find a good one, it will have been dropped in a toilet. It's easier to find good books in a thrift store but you have to have the patience of Job. Stay out of antique stores.