Off road in BBNP is pretty mild insofar as off roading goes. There is not much elevation, no bouldering, hardly any water crossings to speak of (it's in the Chihuahan desert!) . There are some challenging spots but overall it is nothing like Colorado or Utah. I'd almost think driving on I-10 in west Texas at 80+mph might be harder on the CVT than slow crawling in BBNP. High speed driving makes lots of heat.
That said, your modis operandi of putting the selector in manual certainly helps by alleviating the CVT from searching for the best ratio as conditions change.
As for the valve body failure, I would venture that it did not "break" since the valve body itself is but a hunk of metal that hold spool valves, springs, check valves, check balls and solenoids. It does not have much if any structural loads upon it. 30 years ago, The VB
was made of cast iron or steel but I would imagine these days they are aluminum alloy to save weight.
Failure of the valve body could be a spring failure, solenoid failure, leaking spool valve, stuck check ball or valve. Even a gasket leak could cause major problems. Given the way Subaru treats the CVT, I could see the service center simply replacing the entire VB
rather than making a surgical repair.
What could have happened in your situation is the internal filter/screen got clogged by sediment that normally sits in the bottom of the pan but was agitated by the off road bouncing and jostling. That's a theory.
I don't think dust could have worked its way inside the CVT. Although there is a vent tube, dust would clog the tube before enough of it migrated into the transmission to cause problems. The vent tube is bent and curved to make a torturous path for anything trying to work its way in.
Sorry you had to deal with this failure. I do plan to get my OB out to BBNP sometime, so I am glad to see you have been able to negotiate any of the trails out there.
As an ironic anecdote, my closest dealer did not seem interested in doing a CVT fluid change when I asked about it. I will most likely end up doing it myself.