The standard book on African Americans in Seattle is "The Forging of a Black Community: Seattle's Central District from 1870 Through The Civil Rights Era" by Quintard Taylor. It's fascinating and extremely readable.
One of the several things that was exciting for me to learn is that while Seattle's black community was until WWII quite small, it was diverse, and in fact was two separate groups, one, made up of businessmen and their rather posh families, mostly from the East, up at the northern end of the CD centered on the William Grose house up at 28th and Madison, the other lower-class Great Migration southern immigrants ("sharecroppers" as the hoity-toity uptown blacks called them) down at 12th and Jackson. These groups did not always get along too well. In between, the CD grew together as the only part of the city that did not routinely feature racially exclusionary clauses that made it illegal for blacks to settle in, say, Queen Anne or Fremont (Asians and Jews also settled there; the CD has never been an exclusively black ghetto). There's good stuff in here, too, about labor -- which for most of our history has unfortunately been a powerful enemy of racial minorities, from the days of anti-Chinese mobs to the exclusion of blacks from the waterfront and Boeing until surprisingly recently.
Another good book looking at this is Esther Mumford's "Seattle's Black Victorians: 1952-1901".
You should definitely look at "Jackson Street After Hours: The Roots of Jazz In Seattle" by Paul De Barros and Eduardo Calderon for a look at what that "southern" end of Seattle's CD was like when it was hopping with important music. The black club scene depicted in this last book also extends down into the ID, which brings us to....
A good book on the Japanese experience is "Divided Destiny: A History of Japanese Americans in Seattle" by David Takami. You may wonder why the ID is always called the ID, when it looks like Chinatown. That's because for a long time the Japanese were the most significant group there, during the time of the Chinese Exclusion Act, roughly from about 1890 to WWII, when new Chinese immigrants were prohibited and the people who were already here were aggressively chased away. Obviously the shadow of the WWII internment camps hangs over everything related to Japanese-American Seattleites, but it is important to realize that long before that time there was a thriving community there.
In addition to the book, you absolutely MUST visit the Wing Luke Museum, not so much for the exhibits, which are less than stunning, but for the guided tour, which is incredible. Your guide will take you upstairs to some mostly untouched rooms in the hotel upstairs, which illustrate some of the immigration experiences there -- the Chinese, the Japanese, and the Filipino. It's one of the best things in Seattle, and hardly anyone knows about it.
Another thing to see is in the tea room at the Panama Hotel on Main Street. The tea's great, but DO NOT MISS THE MAPS. There are two large hand-drawn wall maps in the back that show block-by-block what was in Japantown, as recalled by old residents, from two time periods, the 1920s and the 1940s. These, along with the old photographs on the wall and the glass plate in the floor, which reveals the undisturbed basement of the hotel as discovered by the new owner, with the trunks of people's belongings left behind when they were taken off to the camps -- it's all powerful stuff.
The relevant chapters in the more general histories -- Murray Morgan's "Skid Road" and Roger Sale's "Seattle Past To Present" top the list.
Another marginalized population living in and out of the mainstream white world is Seattle's gay population, and there's a book about that: "Gay Seattle: Stories of Exile and Belonging" by Gary Atkins. Everybody should read this book! At least the first half of it; later on he gets bogged down in a great deal of radical politics that seemed like it mattered a great deal to certain people at the time but in some ways seems more quaint than the 1850s now. But the first half, which starts off with the earliest recorded appearances of sexual minorities, culled, inevitably, from court cases, well back into the nineteenth century, and then follows their story as it gradually unfolds and stands up for itself, is not just exciting reading in its own right, it's hugely important to the development of Seattle. Seattle was always a strait-laced white city; and its treatment of sexual minorities was part and parcel of its treatment of all excluded people, including not just racial minorities but criminals and partakers of vice (gamblers, pinball players, drinkers, prostitutes; they were all seen as pretty much the same by the police, and treated as such, and this inexorably led to nearly a century of entrenched police and political corruption that eventually exploded into resignations, arrests and prosecutions at the highest city levels -- mayors and police chiefs. It's a stunning story, and it centered around the gay bars. It's told best here.
Another amazing book on early gay Seattle is "An Evening In The Garden Of Allah: A Gay Cabaret" by Don Paulson.
I still haven't found a great book on the Indians who were (and are) here. Not to say one doesn't exist; I just haven't got there yet. They are a constant presence, though, and if you look through the amazing illustrations in Paul Dorpat's "Seattle Waterfront: An Illustrated History", which is not a commercially produced book, but a publication commissioned by the city (and available for checkout at SPL), you will see them as they are slowly pushed out by development.
You also definitely should check out the UW's civil rights web pages, which are a fantastic resource for all of these topics and more: http://depts.washington.edu/civilr/