It is obvious that the jetpack plugin has a bloated code since no one is likely to use all of its modules, but the interesting question is what is the actual impact of the bloat on the site’s performance.
According to tests done by [art of the jetpack team itself, having jetpack active, without even doing anything will delay the time to first byte (TTFB) by about 70 miliseconds taking 470ms instead of about 400ms.
The point here is not the numbers themselves, as it is not very clear what is the setup of the test, but the fact that it is something that actually going to be noticeable, something that actually requires more server resources.
I assume that the problem is with the time it takes for the PHP interpreter to read and interpret the jetpack source files. People that host on VPS can select a host with faster disk access (SSD), have better control of file caching done in memory by the OS, and cache the interpreted code, can and should optimize PHP interpreting aspects. People on shared hosting are just out of luck.
And the usual caveats regarding any performance related discussion apply – if you have most of your pages served from a cache then the impact performance degradation in generating one page is probably much less important to you.