Not likely anything to do with the switch.
There's another component that is involved with controlling the fan speed; it's a resistor, or actually, a series of resistors that are switched by the fan speed control, with different resistors (or resistances) being used at the different 1,2,3 settings. At setting 4, the resistors are out of the circuit and the fan motor is connected directly. The fact that the fan works fine on 4, but not on the other three is a sign that the resistor pack has failed.
I'm not certain how things are laid out in the Impreza, but if it's the same as the Outback, follow the wires coming out of the fan motor (under the dashboard on the passenger side). The wires will go to a connector on the heater duct to the left of where the fan is. That connector is the resistor pack. Remove the connector, and there should be a couple of screws to remove the pack from the duct. Replacements are available at dealers, or some have found them at on-line sources. Install the new one and you should be good to go.
(And do some searching here for fan heater resistor. There are a number of discussions, some with photos.)