> an optimization for e.g. Altivec does not help you at all if you switch to ARM.
The actual code won't help, correct, but AltiVec code could be rewritten into NEON code as a big part of the hard work (identifying vectorizable algorithms, convert scalar code into SIMD code, loop unrolling etc.) has already been done.