There are a limited number of variables that search engines can take into account directly, including keywords, links, and site structure. However, through linking patterns, user engagement metrics and machine learning, the engines make a considerable number of intuitions about a given site. Usability and user experience are "second order" influences on search engine ranking success. They provide an indirect, but measurable benefit to a site's external popularity, which the engines can then interpret as a signal of higher quality. This is often called the "no one likes to link to a crummy site" phenomenon.
When a search engine delivers a page of results to you, they can measure their success by observing how you engage with those results. If you hit the first link, then immediately hit the "back" button to try the second link, this indicates that you were not satisfied with the first result. Since the beginning, search engines have sought the "long click" - where users click a result without immediately returning to the search page to try again. Taken in aggregate over millions and millions of queries a day, the engines build up a good pool of data to judge the quality of their results.
In 2011 Google introduced the Panda Update to its ranking algorithm, significantly changing the way it judged websites for quality. Google started by using human evaluators to manually rate thousands of sites, searching for "low quality" content. Google then incorporated machine learning to mimic the human evaluators. Once its computers could accurately predict what the humans would judge as a low quality site, the algorithm was introduced across millions of sites spanning the Internet. The end result was a seismic shift which rearranged over 20% of all of Google's search results. This caused a dramatic drop in rankings for sites that have performed well in search for years simply because they had poor usability in their site design.
The engines discovered early on that the link structure of the web could serve as a proxy for votes and popularity - higher quality sites and information gained more links than their less useful, lower quality peers. Today, link analysis algorithms have advanced considerably, but these principles still hold true.
We expect the usability elements of websites to continue to grow as an important ranking factor in search engines as the web advances into a more user friendly medium.