fd_set isn't automatically copyable. While it will be possible to fix
that, for now just remove these so that they don't block other changes.