Water flow and water pressure are not the same thing.
Removing your flow restrictor in your shower head would actually reduce the wear and tear on your tub spout diverter, due to less back pressure.
But still, if your water pressure from the city is high, it can wear out the tub spout diverter.
Meanwhile, you may have been buying junky tub spouts from China, so try getting a quality one that has beefier guts inside.
No plastic, heavy solid metal with brass guts with good rubber seals.
And as has already been stated, you will always get some water coming out of the spout, even when it is set to come out of the shower head.
The diverter does not have a complete seal, just more like a 90 or 95 percent closed seal when diverting water to the shower head.