I thought common wisdom said you change all rubber seals before changing to silicone fluid, inferring that swelling only occurs to seals that have previously been subjected to normal brake fluid. That's what I did and a couple of years down the line everything is fine.
That's enough for me. Knowing that all my cylinders and callipers are not rusting up due to water in the fluid is quite a reassurance.