Someone actually came up to me with a question about how we optimized our main site http://stackideas.com . Some has even asked us if Joomla! is really enterprise ready because of the slowness of their site. The answer is really simple, LESS is more. In other words, keep fewer extensions / modules and plugins on your site. Try to go with a ready made solution that just works on the site without requiring the installation of tons of plugins which slows down the site. Even with Wordpress, if you have hundreds of different plugins, it would slow down your site :)
We have optimized our site to only include 5 components and the rest of the Joomla core components has been removed from the site. We have also removed core and additional 3rd party plugins from the site. In essence, here's our current setup and extensions / templates that we have on our site:
These are the only extensions that we have installed on our vanilla Joomla! site and we're extremely happy with the delivery. At times we hit 94 / 100 by Google Page Speed Insights but on an average it hits somewhere around 90 - 94 / 100.
While debugging some of the customer sites, I did notice that there are some that even have up to 200 different plugins and more than 100 different modules. It really doesn't make sense to have so many plugins / modules / components that is installed on your Joomla! site, seriously! Go with a ready made solution for all your needs and minimize the number of Joomla Extensions / Modules and Plugins on your site.