It's true, there can be issues (all of which you can address logically as this guide will show you), there are also some annoyances which come with an early access title modding experience (such as updates changing things which lead to some work on your behalf to correct) but all of this is manageable, not only that, more than worth the time it takes because as you probably already know - modded Bannerlord is proper Bannerlord.
There are some poor souls out there (possibly yourself included at present) who don't dare try modding their game due to the enumerable problems one could face when configuring their load order in a way which will ensure stability throughout their play-through.
So you want to enjoy Bannerlord properly?