Over on Sarah Weinman's blog, thriller reviewer Larry Gandle asked this excellent question, prompting much discussion. I've been meaning to tackle this subject for a while, so here goes.
I rarely have any trouble determining whether or not a book is a mystery or a thriller. The differences between the two are usually evident, although they can be difficult to explain in the abstract.
The reason I think that we tend to get confused is because people so often use the terms interchangeably. Many times, we are told that a book is a thriller, when really it's not.
That's the way it usually works, too, rather than the other way around. Many books are marketed as thrillers that really aren't, because thrillers sell better.
First off: what is a mystery?
In order to be a mystery novel, the story must have a mystery in it. And what's the purpose of having a mystery in a story? For someone to solve it, of course.
Since we're talking about crime novels, the mystery will involve a crime of some sort, which the investigator will attempt to solve. If those two elements are not present in the story, it's not a mystery. (Someone will probably come up with an exception, but I haven't been able to think of any.)
In terms of its format, a mystery novel will work like this: a crime will be introduced near the beginning of the book -- usually a murder, sometimes a theft or similar -- and the rest of the book will be about someone trying to solve that crime.
That "someone" -- the investigator -- can be either a professional or an amateur. S/he can be a detective, cop, private eye, reporter, antiques dealer, bed and breakfast proprietor or anything else.
Can a book be a mystery and also a thriller? Yes, sometimes it happens. A thriller can have a mystery at its core. A thriller can have a romance at its core. Which genre you classify the story as depends on the nature of the story being told, and how the author goes about telling it.
So...what is a thriller?
Thrillers are much looser in form, so it's harder to give as precise a description. You can draw the line at many places, but I think one of the main distinctions lies in motion and emotion.
The plots in thrillers move differently from the plots in mysteries. Mysteries are mono-paced, moving at a steady speed from start to finish. After all, the crime has already happened. The Vicar is lying dead in the chapel; he can't get any more dead.
Thrillers, on the other hand, have greater urgency, as the crime still lies off in the future. Thus, thrillers move more quickly than mysteries, and their pace generally accelerates as the story progresses. A good thriller moves like a roller coaster, with periods of relative calm interspersed with stretches of heightened suspense.
The emotions generated by thrillers are different as well. Mysteries tend to be more intellectual. They excite the mind with their tales of puzzles, investigation, procedure, deduction, analysis and problem solving.
Thrillers are more visceral. They attempt to manipulate our emotions. (Think of Hitchcock and his famous quote about playing the audience like a piano.) Thrillers generate excitement, anticipation, fear, apprehension -- in short, they thrill.
Thrillers come in many forms -- legal, medical, military, action, political, espionage, etc. -- but they all share the common elements of motion and emotion.
One final thing to keep in mind: there are ineffective thrillers. You can read a book and think, "There wasn't one damn thrill in that book -- how can it be a thriller?" In that case, I think it's the book's intentions that determine its classification, even if those intentions failed.