mmm... very strange.
Just now I tried it again, fist few swaps are slow, and after five or six clicks swapping start to go fast. And afer reboot again slow.

But it is not big issue, just for curiosity.
AmigaOS: Micro A1-C, AmigaOne XE, Pegasos II, Sam440ep, Sam440ep-flex, AmigaOneX1000
MorphOS: Efika 5200b, Pegasos I, Pegasos II, Powerbook G4, Mac Mini, iMac G5, Powermac G5 Quad