This is interesting. A few other thoughts on this:

1. Maybe it doesn’t matter too much if players see very little interior art if, as I suspect, GMs are overwhelmingly likely to be the ones actually paying for books. Maybe it’s enough to impress and inspire the ones who are the bulk of the paying audience.

2. Interior art does double duty by improving production values of the product itself and also providing eye catching visuals for traditional and viral marketing (for ads, Kickstarter promotion, store displays, preview images in reviews and on Drivethrurpg, and stuff passed around online—right next to this post in my G+ feed is another post of someone going “check out the interior art from this cool new game”).

3. You are totally right that really evocative art that mismatches the feel of the rules can overpower how the game is played. I’m pretty sure I ran In Nomine differently when all I had was the colorful, cartoonish rules versus the moodier, black and white books. That’s actually why I like reading people’s responses to the question of which book’s art really feels right.

4. I feel like the communication value of visuals applies to design and typesetting even more so than to illustration style, mostly because so many games are so terribly laid out and typeset. That is the main thing that leads to be NOT buy many RPGs that otherwise catch my eye. It’s literally the only thing stopping me from buying a certain sci-fi game supplement with an excellent color illustration on the cover. But maybe that says more about me than about visuals.