Google representatives have stated on several occasions (for an example see this post on the Google Webmaster Central Blog) that the page speed algorithm impacts rankings for less than 1% of search queries.
There have been a few reports from webmasters of page speed significantly impacting rankings :
Keri Morgret from SEOmoz reported that a site with load time issues saw “quite the drop in organic referrals from Google” when the page speed algorithm was pushed live.
Work Coach Cafe reported a 40% growth in traffic after addressing site performance issues.
Smartfurniture.com reported that rankings for 7 of its top 10 keywords improved after site speed was increased.
This data suggests that page speed doesn’t often impact rankings, but when it does the effect can be significant. One plausible interpretation (from Geoff Kenyon) is that “site speed will affect only queries where other ranking signals are very close or when the load time is exceptionally poor.” This interpretation seems to be consistent with Google’s statements and community feedback.
Should SEOs worry about site speed? Page rankings are often based on a combination of dozens of small algorithmic factors; therefore, though page speed is a minor factor, even a small boost could be beneficial for your site.
Don’t obsess over page load speeds, but it would be a good idea to dedicate a small amount of your SEO time and/or budget to speeding up your site. Page speed is also one of the factors totally within your control so its prudent to optimize this ranking factor.