Not a crazy idea, it's actually been tried before:
http://www.reefkeeping.com/issues/20...ture/index.php
Xenia isn't the best for nutrient export, but if your nutrients are too low to support macro, I don't see why it couldn't be used for this purpose. Yes, you would have to harvest/ remove xenia for this to work. One caveat: Xenia consumes trace elements like iodine & mg, so you would want to watch out for that. Maybe have to dose these every once on awhile.
Lots of people keep xenia & sps in the same tank, so I think it'd be feasible.