That is a good point, you can't lower the stopper too much because it does retract slower.
I thought about finding a different bumper that's hard plastic instead, or shaping/coating these in some way to make them more slippery. If they were shaped like a U it might bind less too.
That's it, gotta CNC some aluminum ones