Yeah, RAW doesn’t actually seem to support that approach. I think it’s by design, so that the adventure doesn’t become about the Journey, rather the Journey just tenderizes the meat before you start cooking. 

That said, it’d be trivially easy to turn Hazards into scenes-with-fictional-consequences. You just leave off the mechanical consequences, play the scene out as a scene. Although there’s no actual procedural support for anything that isn’t an Encounter or a Combat, so there’s that. To my eye, that’s an important wrinkle to the system.

The core book play examples themselves, and of course the entirety of Tales, contradicts this — there are plenty of scenes that are “just roleplaying” and not obviously part of the journey/combat/encounter trifecta. So tbh I’m not sure what the designers’ intent is here. I speculate this is another of their developmental blind spots (i.e. everyone knows “how to play an RPG”).