I feel it is possible that the Xbox controller will always be larger in size since the Xbox controller was always kind of large in size since the original Xbox, and most users are used to the large size of the Xbox controller.
I don't think the current controllers for xbox one are too large at all, Nothing like the Duke on the original xbox. That thing was massive and the one reason I didn't buy an xbox back in the day. The 360 controller and now the Xbox One controllers aren't large at all in my opinion, and I got small-ish hands, the xbox 360 and xbox one controllers fit comfortably in my hands though. I prefer it over the PS4 controller, personally.
One of the reasons I became more of an Xbox fan than a PlayStation fan was because of the controller, it's a perfect size or at least has been since the 360 and it is also super popular among Xbox fans as well.
If they were to ever change it, they would have a lot of upset fans.
I don't think that the controller is a large controller. I have big hands, and it fits my hands well. Not only that, but I also think that Xbox is gear more towards adults. Having said that, my 6-year-old grandson doesn't have any problems holding the Xbox Series X controller.
The current Xbox controller for the Series S and X are perfect. They're comfortable to hold, and it could be worse....we could have the controller from the original Xbox as an option still lol.