Google SERP Scraper Guide: Search Results Code Samples

Scraping Google Search results powers rank tracking, SERP monitoring, keyword research, and competitive intelligence. The catch: Google is one of the hardest targets on the web, with aggressive bot detection that blocks raw requests almost immediately. The reliable way to do this in production is through a SERP API that handles unblocking and returns structured results.

The samples below use Bright Data's SERP API, which accepts a normal Google search URL with brd_json=1 and returns parsed JSON (organic results, ranks, links, and snippets) instead of raw HTML.

If Google is your main target, also look at Scrape.do: its Google Search endpoint returns parsed JSON with ads as separate blocks, and in our use it was faster and failed less often than the SERP API. We compare the two in Scrape.do vs Bright Data SERP API.

Prerequisites

Set a PROXY_URL environment variable pointing at your SERP API zone:

export PROXY_URL="http://brd-customer-<id>-zone-<serp_zone>:<password>@brd.superproxy.io:33335"

No account yet? Get started →

PHP

<?php
// Run: php google.php "web scraping tools"
// Requires the bundled cURL + JSON extensions.

$proxy = getenv('PROXY_URL');
$query = $argv[1] ?? 'web scraping tools';

$url = 'https://www.google.com/search?' . http_build_query([
    'q'        => $query,
    'gl'       => 'us',   // country
    'hl'       => 'en',   // language
    'brd_json' => 1,      // ask the SERP API for parsed JSON instead of HTML
]);

$ch = curl_init($url);
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_PROXY          => $proxy,
    CURLOPT_SSL_VERIFYPEER => false, // or install Bright Data's CA certificate
    CURLOPT_TIMEOUT        => 60,
]);

$response = curl_exec($ch);
if ($response === false) {
    fwrite(STDERR, 'Request failed: ' . curl_error($ch) . PHP_EOL);
    exit(1);
}
curl_close($ch);

$data = json_decode($response, true);
foreach ($data['organic'] ?? [] as $r) {
    printf("%d. %s\n   %s\n   %s\n\n",
        $r['rank']        ?? 0,
        $r['title']       ?? '',
        $r['link']        ?? '',
        $r['description'] ?? ''
    );
}

Node.js

// google.mjs — run: node google.mjs "web scraping tools"
// Install: npm i axios https-proxy-agent
import axios from 'axios';
import { HttpsProxyAgent } from 'https-proxy-agent';

const agent = new HttpsProxyAgent(process.env.PROXY_URL);
const query = process.argv[2] ?? 'web scraping tools';

const url = 'https://www.google.com/search?' + new URLSearchParams({
  q: query, gl: 'us', hl: 'en', brd_json: '1',
});

const { data } = await axios.get(url, {
  httpsAgent: agent,
  proxy: false,        // route everything through the agent
  timeout: 60_000,
});

for (const r of data.organic ?? []) {
  console.log(`${r.rank}. ${r.title}\n   ${r.link}\n   ${r.description ?? ''}\n`);
}

Rust

// Cargo.toml:
//   reqwest = { version = "0.12", features = ["blocking", "json"] }
//   serde_json = "1"
use serde_json::Value;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let query = std::env::args().nth(1).unwrap_or_else(|| "web scraping tools".into());

    let client = reqwest::blocking::Client::builder()
        .proxy(reqwest::Proxy::all(std::env::var("PROXY_URL")?)?)
        .danger_accept_invalid_certs(true) // or install Bright Data's CA
        .build()?;

    let data: Value = client
        .get("https://www.google.com/search")
        .query(&[("q", query.as_str()), ("gl", "us"), ("hl", "en"), ("brd_json", "1")])
        .send()?
        .json()?;

    if let Some(results) = data["organic"].as_array() {
        for r in results {
            println!(
                "{}. {}\n   {}\n   {}\n",
                r["rank"],
                r["title"].as_str().unwrap_or(""),
                r["link"].as_str().unwrap_or(""),
                r["description"].as_str().unwrap_or("")
            );
        }
    }
    Ok(())
}

What the samples return

With brd_json=1, the SERP API returns parsed JSON instead of HTML. Our SERP API review lists the main blocks:

The samples print only the organic results. Print the whole response once to see every field for your query and country.

When results come back empty

• No organic key. Some queries return mostly ads, maps or other blocks. Print the full JSON before assuming the request failed.
• Results from the wrong country. Google personalises by location. Set gl (country) and hl (language) in the URL, as the samples do, and keep them fixed across runs so rankings stay comparable.
• An authentication error. Check that PROXY_URL uses the SERP API zone's credentials, not another zone's.

Notes

• The SERP API returns far more than organic results — data also includes ads, related_searches, people_also_ask, and pagination. Inspect the full JSON to see every field.
• For rank tracking, run this on a schedule and store each result's rank per keyword over time.
• Want raw HTML instead of JSON? Drop the brd_json parameter and parse the markup yourself (selectors change often, which is why JSON is recommended).

Staying within the rules

Collect public search results only, keep request rates modest, and read the site's terms before you build anything commercial on the data. Don't scrape account pages, personal details, or anything behind a login. Our how to avoid getting blocked guide covers polite crawling in more detail.

FAQ

Why use a SERP API instead of scraping Google directly?

Google blocks plain scrapers quickly, and its result markup changes often. A SERP API handles the unblocking and returns parsed JSON, so you maintain neither proxies nor parsers. The SERP API review covers how Bright Data's version bills only for successful requests.

How do I get results for a specific country?

Set the gl parameter to the country code and hl to the language, for example gl=us&hl=en. Keep them the same on every run if you're tracking rankings over time.

Can I get the raw HTML instead of JSON?

Yes. Leave out brd_json=1 and you get the page's HTML. You then have to maintain your own parser, which is why the JSON output is recommended.

Struggling with blocks on your own setup? Read How to Avoid Getting Blocked While Web Scraping and our Bright Data SERP API review.

Get started with Bright Data →