Note: I'm not sure this is a bug, but it's also not really a suggestion. It's more a request to maybe complete the development of a partially completed feature, or at least document it is incomplete in man pages and relevant docs.
zpool remove of top level vdevs is a significant new feature. But at the moment, it seems only partially implemented.
- Requires entire pool (not just vdev being removed) to have mirrors only not raidz
- Requires entire pool (not just vdev being removed) to have same sector sizes
- Even when disks are all mirrors and have same sector sizes, may refuse to work. (Tested on a pool of mirrored 4096 logical/512 physical HDDs, sector sizes confirmed in console)
That's a problem if a user believes top level vdevs can be removed now, and finds they can't - but only discovers it at removal.
Since there's no other way to remove a top level vdev short of replicating the pool, this is a concern.
And also, what's with the demands on other vdevs? Nothing else in ZFS demands that vdevs have any specific sector or structure type, when copying data. Mostly, ZFS is transparent as to on-disk physical/logical sector sizes, or mirror vs raidz; it's just block sizes that count.
Can we lose these 2 limitations, so that any top level vdev in any pool can be removed, not just some of them?