I think I have it figured out @martindb. You are using the “Lead in feed rate” rather than the “plunge feed rate”.
My understanding is “Lead in” is from the side as it enters the work piece. This is useful to slowly load the bit before accelerating to the cutting feedrate. “Plunge” is from the top down into the material like a drill bit.
Most of my moves should actually be a plunge, lead in, cut, lead out, travel z, move, travel z, repeat. For the attached tool path anyway.