I have not tested it with the steam deck specifically but when I used it on another Linux device it was pretty much plug and play. If you do go this route make sure you get a genuine adapter. There is a lot of clones on amazon.

Official Xbox series controllers are relatively cheap for their quality (~40 dollars on sale) and there is an official Xbox dongle that allows for the connection of I think up to 8 controllers with a pairing button. I have not tested this configuration extensively but I think the dongle handles a lot of the logic so it should work better with worse drivers. Also when setting up everything one must just press the pairing button on the dongle and on each controller to get everything setup.