In one of my apps based on the basic template, the vendor directory weighs more than 40 MB. But the app runs lightening fast.
Thanks to the autoloading mechanism, Yii will load only the necessary code into the memory from the vendor directory. The installed extensions usually have a lot of non-runtime stuff like documentation and code for development. They don’t affect the performance of your app, not at all.