How much of the Internet is indexed by Google?
Despite Google’s immense index, it’s widely understood that its search tools only index a fraction of the total internet. The vast majority of the web is known as the deep web, which includes content behind paywalls, password-protected pages, and dynamic content that isn’t publicly linked. The part of the deep web that is intentionally hidden is called the dark web, which is a very small part of the total internet.
Instead of manually searching through every page for keywords, search engines like Google use a specialized data structure called an inverted index (also known as a reverse index) to quickly find relevant information.
Inverted Index and the Semantic Web
A forward index maps each document to the words it contains, which is inefficient for searching. An inverted index, on the other hand, is a more efficient data structure that maps each term to the documents where it appears. This allows a search engine to find all documents containing a specific word with just a single lookup.
To create this inverted index, the text of each document is first preprocessed using techniques to normalize the text:
- Lemmatization reduces words to their base form (e.g., “running” and “ran” are both converted to “run”). This is more linguistically accurate than stemming, which simply cuts off the end of a word and can sometimes produce non-words.
- Removing stop words gets rid of common but contextually meaningless words like “the,” “of,” and “and,” which helps reduce the size of the index.
Today’s crawling and indexing go far beyond just keywords. Search engines also look for semantic triples in the form of subject-predicate-object statements. These triples represent factual relationships between entities and form the building blocks of the Knowledge Graph. For example, a page might contain the triple: “Marie Curie” (subject) “discovered” (predicate) “radium” (object). By identifying these triples, Google understands the meaning and context behind the text, not just the words themselves.
How Knowledge Graphs and Panels are Created
The Knowledge Graph is Google’s massive database of billions of facts and entities—people, places, things, and the connections between them. This graph is built by crawling and indexing not just keywords, but the semantic triples found in structured data (like Schema Markup), Wikipedia, and other authoritative sources.
The knowledge panel is the direct result of this process. It is the information box that appears on the right side of a desktop search results page (or at the top of a mobile page) when you search for a specific entity. The panel is a summary of the most important facts about that entity, all pulled directly from the Knowledge Graph. For example, a search for “Mount Everest” might bring up a knowledge panel with its elevation, location, first climbers, and a picture.
The existence of a knowledge panel for a person or entity is a strong signal that Google has a high degree of confidence in its understanding and trustworthiness of that subject. For content creators, providing clear, consistent, and structured data through semantic triples is the primary way to contribute to the Knowledge Graph and increase the chances of getting a knowledge panel.
How Google Ranks Results Today
When a user types a query, Google’s system finds all the documents that contain those words in its index and retrieves the relevant facts from its Knowledge Graph. The real challenge, and the core of Google’s algorithm, is to rank those documents by relevance. This is done using hundreds of ranking factors, which can be grouped into a few key areas:
- Relevance: Factors that determine how closely a page’s content matches the user’s query, including keywords in the title, headers, and body text.
- Backlinks and Authority: The original PageRank algorithm still exists, but it has been heavily updated. It evaluates the quantity and quality of links pointing to a web page. A link from a highly trusted site is valued far more than a link from a low-quality site.
- E-E-A-T (Experience, Expertise, Authoritativeness, and Trustworthiness): This is a critical quality guideline that Google uses to evaluate content, especially for topics that could impact a person’s health or finances. It assesses whether the content creator has real-world experience, a strong foundation of expertise, is recognized as an authority on the topic, and is fundamentally trustworthy.
- User Experience: Google also considers how users interact with a page, including its loading speed, mobile-friendliness, and overall ease of use. This is measured by metrics like Core Web Vitals.
By combining all these factors, Google’s system scores each potential result and presents the most relevant and highest-quality pages at the top of the search results in a matter of milliseconds.
