how search engines work

How Search Engines Work

How Search Engines Work

In order for us to understand how to get your page visible to viewers we need to understand how search engines work. The biggest piece of the SEO puzzle is figurign out how to get your site seen and show up in the Search Engine Results Page (SERP).

Search engines have 3 primary functions:

  1. Crawl- Scour the web for content.
  2. Index- Store and organize found content.
  3. Rank- Provide the best pieces of content that answer the search query.


Search Engine Crawling

 There are plenty of content sources out on the web. These can be web pages, PDF files, images and videos. The search engine will send out a group of robots, Sometimes called crawlers or spiders, to scour the websites out there for new and updated content.

These crawlers will start with a few pages then branch out by following links on those pages to find URLs to new pages. They create a massive database that is used for when a user starts a search of a specific topic.

Search Engine Indexing

The index is the organization of the massive database that the search engine has developed through crawling. By organizing the data into indexes it cuts down the time it would take to finish a search significantly.

The crawlers keep the database up-to-date to maintain the reliability of the index. There are many factors that come into creating the index, such as design structures and data structures but that is information we do not need to cover.

Search Engine Ranking

After finding the sites with information most relevant to the search query the engine orders the results to be displayed on the SERP. This is known as ranking. Overall, the higher in the SERP the website is ranked, the more relevant it is to the query searched by the user.

Though there are several different search engines out there including Bing, Yahoo and others, the majority of developers focus on Google. Google is used by the vast majority of people when searching for something on the web. This could be web searches, images or even Google Maps searches. More than 90% of searches are made using Google. Even Youtube is a product of Google.

Crawling: Make Sure Search Engines Can Find Your Pages

Ensuring that the search engine crawlers can even find your site is a critical first step of SEO. One way to find out if the crawlers can find your pages is by going to search and using “”. This will display which of your web pages will even be accessible when crawling.

Some pages can be hidden from the crawler robots and can not be found. There are several reasons this may happen. Some reasons your site can not be found by crawlers are:

  • Your site is too new and hasn’t been crawled yet.
  • Your site has no external links from other websites.
  • Your sites navigation is improperly structured.
  • Your site could be penalized by Google for spammy actions.

Sometimes the search engine is unable to find some of your sites important content or URLs. If your site requires a log in to access certain information then the search robot will not be able to access that page.

Some developers rely on a search algorithm and think by adding a search bar that any user would be able to find what they need. This not only hinders the users experience on your web page but also limits the crawlers attempting to navigate your site. If there aren’t links or an easily structures site architecture than the crawlers will not be able to find all of your content.

Indexing: How Search Engines Store Your Pages

Just because your site can now be crawled doesn’t mean it can be indexed. After a crawler finds a page the search engine would open it like a browser would, then searches through the information on the page to figure out where to store the information. The way this information is stored is the index.

Pages are searched and cached in different frequencies based on the popularity of the site and how often the site updates or publishes posts. For example, will be crawled much more frequently then say a local business website for a towing company or restaurant.

Pages can also be removed from an index. Several reasons can cause this to happen. For example, if your site has a noindex meta tag added or if the site requires a password to reach a URL your site could be removed from the index.

Ranking: How Search Engines Rank Your URL

The process of listing search results of a query by most relevant to least relevant is known as ranking. This is how the search engine, like Google, ensures the searcher finds the best information for what they’re searching for.

Search engines use an algorithm to sort the most relevant information to be presented to the user. These algorithms are constantly updated but are to make the search engines more effective in delivering results. Google updates their algorithm every day but the updates are usually tweaks here and there until a larger update is created.

If your site has taken a dive due to an update or you are trying to keep up-to-date on the algorithm you can check out Google’s Quality Guideline or Search Quality Rater Guidelines.

These algorithms are effective now but used to be rudimentary in the past. This caused pages to over spam their sites with keywords or make them bold in order to rank higher on Google. This doesn’t work anymore and is frowned upon. More on keyword use here.

Links and SEO

Backlinks or “inbound links” are links on a page that lead to your website or vice versa. Internal links are links on your page that lead to another one of your pages. The more natural backlinks your site has from other websites acts like “word of mouth” and creates rapport for your site.

Google has a part of its algorithm called PageRank which uses backlinks and internal links to rank the authority and relevance of your site. Bottom line, the more natural backlinks exist to your site the better your site will rank.

Content and SEO

Links are very helpful but are meaningless if they lead to an irrelevant or empty URL. Content can be anything from text on a page, images, or videos. The search engine searches the content on your page to determine its relevance to the present search query.

Search engines are also focused on the user and their satisfaction. Because of this the search engine will search for words that are relevant to their search. This includes “alt text” in images and videos or any other content available on the page.

Engagement Metrics: Click Data

Another part of SEO is the actual data of how users engage your site. Some believe high engagement is caused by good SEO and some believe good engagement causes good SEO. It is clear that how your users engage your site will improve your SEO.

Several elements factor into User Engagement:

  • Clicks (visits from a search)
  • Amount of time the user spends on your page before leaving
  • Bounce rate (percentage of visits where users only viewed on page on your site before leaving)
  • Pogo-Sticking (clicking on an organic result then quickly bouncing back to the SERP to click on another result)

Since user engagement can make a ranking effect it is important to create your page to attract a user and keep them engaged in your site.

Local Business and SEO

If the user is searching for a local business, some search engines like Google have their own proprietary index for those searches. Google’s feature for these indexes is Google My Business. You can learn more on Google My Business and optimizing your business listing here.

Google uses 3 main factors when utilizing local searches:

  • Relevance
  • Distance
  • Prominence

Relevance is how well the business the user is searching for matches the search query. To make sure your Google My Business listing is relevant in searches make sure your information is filled out completely and accurately.

The distance factor pertains to your geo-distance to the surrounding businesses being searched. Google places emphasis on your location with location services with Google Maps.

Prominence factor is Google wanting to reward businesses in the real world. This includes how the business performs online and offline as well as some other factors.

Now more than ever, search influences are being influenced by real world successes and applications. This could be user reviews or links etc.

Next we are going to focus on Keyword Research.

If you are looking for more information on setting up Google My Business check out Importance of Google My Business by S & S Business Consultants.

Comments are closed.