GeezerVison

Google Custom Search Engine (CSE) is a search engine that is suited for developers in which it lets you include a search engine in your application, whether it is a website, a mobile app, or anything else.

Since manually scraping Google Search is highly unsuggested, as it will restrict with a reCAPTCHA every few queries, in this tutorial, you will learn how you can set up a CSE and use its API in Python.

In CSE, you can customize your engine that searches for results on specific websites, or you can use your website only. However, we will enable our search engine to search the entire web for this tutorial.

Learn also: How to Use Google Drive API in Python.

Setting Up a CSE

First, you need to have a Google account to set up your search engine. After that, head to the CSE page and sign in to Custom Search Engine as shown in the following figure:

After you log in to your Google account, a new panel will appear to you that looks something like this:

You can include the websites you want to include in your search results, choose the language of your search engine, and set up its name. Once finished, you'll be redirected to this page:

Using CSE API in Python

Now to use your Search Engine in Python, you need two things: First, you need to get your Search Engine ID, you can get easily find it in the CSE control panel:

Second, you have to generate a new API key, head to the Custom Search JSON API page, and click on the "Get a Key" button there, a new window will appear, you need to create a new project (you can name it whatever you want) and click on Next button, after that you'll have your API key, here is my result:

Lastly, if you want to search the entire web, you need to enable it on your control panel:

Now you have everything you need to use the CSE API in your Python code, open up a new Python file and follow along. We'll be using the requests library for convenience. You can install it using this command:

pip3 install requests

Let's initialize our CSE requirements:

import requests

# get the API KEY here: https://developers.google.com/custom-search/v1/overview
API_KEY = "<INSERT_YOUR_API_KEY_HERE>"
# get your Search Engine ID on your CSE control panel
SEARCH_ENGINE_ID = "<INSERT_YOUR_SEARCH_ENGINE_ID_HERE>"

We will do a simple search query, "python" for demonstration purposes. Let's build the API URL we'll request:

# the search query you want
query = "python"
# using the first page
page = 1
# constructing the URL
# doc: https://developers.google.com/custom-search/v1/using_rest
# calculating start, (page=2) => (start=11), (page=3) => (start=21)
start = (page - 1) * 10 + 1
url = f"https://www.googleapis.com/customsearch/v1?key={API_KEY}&cx={SEARCH_ENGINE_ID}&q={query}&start={start}"

Here we're building the URL that we will request. By default, the CSE API returns the first ten search results. Changing the page number to 2, for instance, will make start API parameter set to 11, so it'll return the second-page result, and so on.

Making the API request and using the requests' json() method to automatically parse the returned JSON data to a Python dictionary:

# make the API request
data = requests.get(url).json()

Now, this data is a dictionary containing many result tags. We are only interested in "items", which are the search results. By default, CSE will return ten search results. Let's iterate over them:

# get the result items
search_items = data.get("items")
# iterate over 10 results found
for i, search_item in enumerate(search_items, start=1):
    try:
        long_description = search_item["pagemap"]["metatags"][0]["og:description"]
    except KeyError:
        long_description = "N/A"
    # get the page title
    title = search_item.get("title")
    # page snippet
    snippet = search_item.get("snippet")
    # alternatively, you can get the HTML snippet (bolded keywords)
    html_snippet = search_item.get("htmlSnippet")
    # extract the page url
    link = search_item.get("link")
    # print the results
    print("="*10, f"Result #{i+start-1}", "="*10)
    print("Title:", title)
    print("Description:", snippet)
    print("Long description:", long_description)
    print("URL:", link, "\n")

We are extracting the title, long description, the snippet (i.e., description), and the link to the resulting page. Check the result:

========== Result #1 ==========
Title: Welcome to Python.org
Description: The official home of the Python Programming Language.
Long description: The official home of the Python Programming Language
URL: https://www.python.org/

========== Result #2 ==========
Title: The Python Tutorial — Python 3.8.2 documentation
Description: It has efficient high-level data structures and a simple but effective approach to 
object-oriented programming. Python's elegant syntax and dynamic typing,
together ...
Long description: N/A
URL: https://docs.python.org/3/tutorial/

========== Result #3 ==========
Title: Download Python | Python.org
Description: Looking for Python with a different OS? Python for Windows, Linux/UNIX, Mac OS     
X, Other. Want to help test development versions of Python? Prereleases ...
Long description: The official home of the Python Programming Language
URL: https://www.python.org/downloads/

That's awesome, I have snipped the output to only three search results, but it will print ten results in your case.

Conclusion

You can specify various request parameters to customize your search. You can also print the data dictionary to see other metadata, such as total results, search time, and even the meta tags of each page. Check CSE's documentation for further details.

By default, it is limited to 100 search queries per day for free access. If you feel that isn't enough for your application, then consider signing up for billing in the API Console.

A great use case of this API is to get the Google ranking of a specific page by a specific keyword. Check this tutorial in which I show you how!

Now you can integrate searching techniques into your applications, and I hope this tutorial was helpful and easy to follow. If you like it, please don't hesitate to share it!

There are other tutorials on Google APIs. Here are some of them if you want to check them out:

• How to Extract Google Trends Data in Python.
• How to Use Google Drive API in Python.
• How to Extract YouTube Data using YouTube API in Python.
• How to Use Gmail API in Python.

Read also: How to Make a URL Shortener in Python.

Happy Coding ♥