Google uses machines to scan the web and return relevant results for your search queries. Humans don’t often get involved in that process, but there are some things machines still aren’t good at. For example, evaluating the “quality” of a source. That’s why Google has a small army of humans around the world checking search results for quality. Now, these “quality raters” have been given a new directive — to flag offensive and upsetting content in searches. It turns out machines aren’t great at figuring out what humans will find offensive.
The existence of quality raters was first made public in 2013 when Google released the manual they use to flag search results. When provided with search data, raters can mark results using various criteria like spam, porn, or illegal. This helps Google adjust its algorithms to remove results and improve the links and Knowledge Graph data it shows to users. Individual flags aren’t used to change results. Google is constantly making changes to its algorithm, and any change the results in fewer flags is marked as positive.
While Google has not commented on the addition of the offensive-upsetting flag, it’s probably related to the increase in overtly racist content that’s being juiced up specifically to appear in search results. That’s the dark side of search engine optimization. In one particularly troubling incident a few months ago, searches on Google for “did the Holocaust happen” were showing a link to the white supremacist forum Stormfront at the top. That article gave people tips on how to explain and promote Holocaust denial. That’s probably not the appropriate thing to show at the top of search results.
The above scenario is the example used in Google’s new quality rating manual for the offensive/upsetting flag. The link to Stormfront is not flagged as porn, foreign language, or didn’t load. It is, however, flagged as offensive. The rationale given is that holocaust denial is linked to antisemitism, and is therefore offensive to many people. A different result on the Holocaust from the History Channel is not marked as offensive because it gives accurate, well-sourced information on the events.
Even before the introduction of the offensive flag, results like this were often demoted for quality reasons. The website might be slow, running sketchy ads, or the content may simply have been poorly presented. The new flag gives Google’s quality raters a simple catch-all for content that needs to be kept out of the top slots.