There are two separate fixes here:
1) vfs_streams_xattr uses get_ea_value() from source/smbd/trans2.c,
which imposes a configurable limit on max xattr size. This is reasonable
because we cannot use pread() / pwrite() for xattrs. The max xattr size
was not being checked during write() requests, which could result in
xattrs being written which were larger than could be read through
get_ea_value(). Impose same limit in vfs_streams_xattr. Return -1 with
errno set to EOVERFLOW in case xattr write is larger than permitted.
Add special handling for this errno in smb2 write path so that we return
STATUS_FILE_SYSTEM_LIMITATION to the SMB client. This behavior matches
observed behavior of Microsoft's ReFS when overly-large ADS write is
attempted over the SMB protocol. In this case File Explorer will report
that it encountered an issue with a filesystem limitation and provide
option to skip the file.
2) If large xattr is written to filesystem and does not have the special
streams_xattr prefix on it (or if streams_xattr is configured to store
extended attributes in a way compatible with Netatalk), then SMB clients
will be presented with error on EA (non-ADS) reads. This is due to protocol
limitation on size of EAs. During marshalling of EA list for SMB response,
check EA size and skip any over 64KiB in size.