01 VDC, 05 R Sedan, 06 BAJA EJ257
Dito on the engine braking and comparison. The TCM keeps the TCC locked to assist in braking. You could be feeling the TCC unlocking which changes the rpm as there is a lapse in idle control.I can recall posts, particularly by new-to-Subaru owners, with this same concern, and wonder if this is what you're referring to. This isn't to say that what you're experiencing can't be something different, but I don't have a good sense of what your "shift hard" feels like.
As already mentioned, roughness can be one of the symptoms that might be experienced when Type HP ATF isn't used. However, the normal 2 to 1 downshift in the 4EAT is, nevertheless, noticeable. The transmission's electronic-control programming is designed to make use of engine braking and this exaggerates the downshift effect, especially with the tachometer visibly jumping up and the physically-sensed rate of deceleration increasing a notch just as the car approaches the stopping point.
I'm not saying you can't have a problem, but if you haven't driven similar Subarus before, it might be worthwhile doing so now for comparison. Then you'd have a better idea if your symptoms are different. Perhaps your mechanic would allow you to come along when test driving cars after servicing, or if there's similar Subaru's in your area, discussions, and even demonstration rides, with their owners could provide confirmation of a problem, or hopefully, peace of mind.
If it is shifting fine up through the gears and you are not experiencing any noise or vibration, unless it is a loud hard thunking, I wouldn't be concerned with it. If there is a thunking, you may want to look into mounts, but that would be unlikely considering the mileage.
There aren't any TSB's as yet concerning this, so Subaru hasn't seen a wide range of issues. The only shift related TSB is the commonality of harsh shift in cold weather which holds 1st gear longer for warmup and this is just the way the TCM is programmed.