Social Media Scraping: Profiles, Posts & Sentiment

Brands, PR agencies, political researchers, and financial analysts all need programmatic access to the same thing: what is being said publicly on social media, and by whom. Sentiment tracking, influencer discovery, trend monitoring, brand-safety auditing — every one of these workflows starts with collecting public profile data, post text, and engagement signals at a pace that no manual process can sustain.

This guide covers social media scraping end to end: what public data is accessible without a login, why social platforms are among the hardest scraping targets on the web, the code patterns that work reliably across Instagram, TikTok, and Facebook, and how to architect a sentiment-monitoring pipeline that stays resilient. The per-platform guides linked throughout go deeper on site-specific parsing.

Social scraping falls into three tiers, each with different technical requirements and different sensitivity profiles.

Public Profile Pages

Every major platform renders a profile page to anonymous visitors: display name, bio, follower count (where disclosed publicly), post count, and a grid or feed of recent public content. This page is the most stable scraping target because it is designed to be indexable by search engines — which means it reliably includes Open Graph meta tags and, on some platforms, a JSON-LD ProfilePage or Person block.

Full technical guides with PHP, Node.js, and Rust code:

How to Scrape Instagram — public profiles, posts, and hashtag pages
How to Scrape TikTok — public video feeds, creator stats, and trending audio
How to Scrape Facebook — public pages and marketplace listings

Post and Hashtag Feeds

Individual post pages carry captions, hashtags, timestamps, and engagement counts (likes, comments, shares where public). Hashtag and topic pages aggregate recent public posts under a tag — useful for trend detection and brand-mention monitoring. Both require JavaScript rendering on TikTok and Facebook; Instagram's post pages serve partial data server-side via Open Graph.

Engagement Signals and Sentiment Indicators

Like counts, comment counts, and share counts — where platforms make them public — are the raw ingredients for sentiment scoring. They tell you which posts resonate, and tracking those signals over time reveals whether sentiment around a brand, topic, or creator is accelerating or decaying. Comment text from public posts carries the richest qualitative signal but requires pagination and rate-aware crawling to collect at scale.

Social media sites share a cluster of defences that make them consistently difficult:

JavaScript rendering is mandatory for most pages. TikTok and Facebook are React SPAs. The HTML shell the server returns contains no useful content — all profile data, post metadata, and engagement numbers appear only after client-side hydration. A plain HTTP client returns a loading spinner, not data.

CDN-level IP filtering is aggressive. All three platforms reject datacenter and cloud provider IP ranges wholesale at the network edge, before any application logic runs. Residential or ISP proxies with realistic TLS fingerprints are the baseline requirement, not an optimisation.

Bot scoring is multi-signal. Instagram's and Facebook's bot-detection systems correlate request cadence, referer chains, cookie state, browser fingerprint, and behavioural timing in combination. A scraper that passes the IP check can still hit CAPTCHA walls or soft-blocks if any other signal looks programmatic.

Markup changes silently. Unlike e-commerce sites where product schema is relatively stable, social platforms push markup updates continuously. An embedded window._sharedData key that works this week may be removed next week with no announcement. Open Graph meta tags survive layout changes far better than internal JavaScript state objects.

See How to Avoid Getting Blocked and Proxy Types Explained for detailed countermeasures on each front.

Prerequisites

All code in this guide routes requests through the Bright Data Web Unlocker, which handles JavaScript rendering, residential IP rotation, TLS fingerprint normalisation, and CAPTCHA resolution behind a single proxy endpoint:

export PROXY_URL="http://brd-customer-<id>-zone-<unblocker_zone>:<password>@brd.superproxy.io:22225"

No Bright Data account yet? Explore Bright Data's social media data tools →

Extracting Open Graph Metadata from Public Profiles

Open Graph meta tags are the most stable parsing target across all social platforms. Instagram, TikTok, and Facebook all emit og:title, og:description, og:image, and og:url on every public-facing page — these tags are required for link previews and change far less often than internal JavaScript state objects. The samples below parse Open Graph tags from any public social profile URL and serve as the base pattern before you layer on the site-specific parsing shown in the per-platform guides.

PHP

<?php
// Run: php social.php "https://www.tiktok.com/@username"
$proxy = getenv('PROXY_URL');
$url   = $argv[1] ?? 'https://www.tiktok.com/@username';

$ch = curl_init($url);
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_PROXY          => $proxy,
    CURLOPT_SSL_VERIFYPEER => false,
    CURLOPT_TIMEOUT        => 60,
    CURLOPT_HTTPHEADER     => [
        'Accept-Language: en-US,en;q=0.9',
        'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    ],
]);
$html = curl_exec($ch);
curl_close($ch);

$doc = new DOMDocument();
@$doc->loadHTML($html);
$xp = new DOMXPath($doc);

$og = [];
foreach ($xp->query('//meta[starts-with(@property,"og:")]') as $node) {
    $key        = str_replace('og:', '', $node->getAttribute('property'));
    $og[$key]   = $node->getAttribute('content');
}

// Also capture twitter:description as a fallback for richer bio text.
$twitterDesc = $xp->query('//meta[@name="twitter:description"]')->item(0);
if ($twitterDesc) {
    $og['twitter_description'] = $twitterDesc->getAttribute('content');
}

echo json_encode(['url' => $url, 'og' => $og],
    JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE), PHP_EOL;

Node.js

// social.mjs — node social.mjs "https://www.tiktok.com/@username"
// Install: npm i axios https-proxy-agent cheerio
import axios from 'axios';
import { HttpsProxyAgent } from 'https-proxy-agent';
import * as cheerio from 'cheerio';

const agent = new HttpsProxyAgent(process.env.PROXY_URL);
const url   = process.argv[2] ?? 'https://www.tiktok.com/@username';

const { data: html } = await axios.get(url, {
  httpsAgent: agent, proxy: false, timeout: 60_000,
  headers: {
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  },
});

const $ = cheerio.load(html);
const og = {};

$('meta[property^="og:"]').each((_, el) => {
  const key = $(el).attr('property').replace('og:', '');
  og[key]   = $(el).attr('content');
});

// twitter:description often carries a longer bio than og:description.
const twitterDesc = $('meta[name="twitter:description"]').attr('content');
if (twitterDesc) og.twitter_description = twitterDesc;

console.log(JSON.stringify({ url, og }, null, 2));

Rust

// Cargo.toml:
//   reqwest = { version = "0.12", features = ["blocking"] }
//   scraper = "0.20"
//   serde_json = "1"
use scraper::{Html, Selector};
use serde_json::{json, Map, Value};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let url = std::env::args().nth(1)
        .unwrap_or_else(|| "https://www.tiktok.com/@username".into());

    let client = reqwest::blocking::Client::builder()
        .proxy(reqwest::Proxy::all(std::env::var("PROXY_URL")?)?)
        .danger_accept_invalid_certs(true)
        .build()?;

    let html = client
        .get(&url)
        .header("Accept-Language", "en-US,en;q=0.9")
        .header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
        .send()?
        .text()?;

    let doc  = Html::parse_document(&html);
    let sel  = Selector::parse(r#"meta[property]"#).unwrap();
    let tsel = Selector::parse(r#"meta[name="twitter:description"]"#).unwrap();

    let mut og = Map::new();
    for el in doc.select(&sel) {
        let prop = el.value().attr("property").unwrap_or("");
        if prop.starts_with("og:") {
            let key = prop.trim_start_matches("og:");
            let val = el.value().attr("content").unwrap_or("");
            og.insert(key.to_string(), Value::String(val.to_string()));
        }
    }

    if let Some(tw) = doc.select(&tsel).next() {
        if let Some(c) = tw.value().attr("content") {
            og.insert("twitter_description".into(), Value::String(c.to_string()));
        }
    }

    println!("{}", serde_json::to_string_pretty(&json!({
        "url": url,
        "og": og,
    }))?);
    Ok(())
}

The per-platform guides extend this foundation with site-specific structured data: Instagram's JSON-LD ProfilePage block, TikTok's embedded __UNIVERSAL_DATA__ state object for video-level statistics, and Facebook's page-state JSON for page-type metadata.

Architecture for a Sentiment-Monitoring Pipeline

A social sentiment monitor has the same scaffold as any other time-series tracker, with a few domain-specific layers on top:

Seed your watchlist from profile pages. Collect canonical profile URLs for the brands, creators, or topics you are tracking. Open Graph data gives you the og:url to canonicalise each handle across platform URL formats.
Parse the stable metadata first. Open Graph tags survive layout changes. Extract those before attempting embedded JavaScript state; log failures per-source so markup drift shows up as a structured field-null spike rather than a silent gap.
Paginate post feeds with session continuity. Scraping a feed beyond the first page typically requires a cookie jar and referer chain that mirrors organic browsing. Rate-limit aggressively — most platforms throttle well before they block if cadence looks human.
Store time series, not just snapshots. Append {profile_id, post_id, likes, comments, scraped_at} on every run. Sentiment value comes from the trajectory: a post gaining 10,000 likes in an hour reads very differently from one gaining 10,000 likes over a week.
Alert on null-rate spikes. Track the fraction of requests where key fields come back null per platform. A jump to near-100% null on TikTok but not Instagram or Facebook means TikTok changed its markup — not that all creators went dark.
Apply entity tagging at ingest. Identify brand mentions, product names, and competitor references in caption text at ingest time. Storing raw text with entity tags attached makes downstream sentiment scoring much faster to query.

For detailed guidance on change detection and storage patterns, see the E-commerce Scraping overview — the tracker architecture transfers directly.

Legal and Ethical Considerations

Social scraping of public data sits in a well-documented grey zone. The data is visible to any anonymous browser; the platforms' Terms of Service uniformly prohibit automated collection. Those are separate questions with separate consequences:

Public data only. Profile information and posts visible without logging in are categorically different from anything behind an authentication wall. Never authenticate to scrape; logged-in scraping crosses a clear line under the Computer Fraud and Abuse Act (US) and comparable statutes elsewhere.
Personal data rules apply even to public posts. Post text, profile photos, and display names may constitute personal data under GDPR or CCPA even when publicly posted. Apply appropriate retention, minimisation, and purpose-limitation policies.
Throttle to avoid harm. Request rates that degrade platform performance for real users create legal exposure and accelerate blocks. Scrape at a cadence that is invisible in aggregate traffic.
Get jurisdiction-specific legal advice. The legality of scraping public social data under the CFAA, GDPR, and platform ToS evolves through ongoing court decisions. Don't rely solely on this guide — get qualified legal counsel before deploying a production scraper.

Scaling to Production

Monitoring hundreds of creator profiles, tracking hashtag velocity across multiple platforms, or running brand-safety audits at scale requires infrastructure that compounds quickly: residential IP rotation, JavaScript rendering capacity, CAPTCHA handling, per-platform cookie management, and markup-change monitoring. Bright Data's social media data collection tools handle that infrastructure layer, returning clean structured data without you managing proxy pools or headless browser fleets. For bulk historical data needs, pre-built datasets are often faster to acquire than bootstrapping a production scraper from scratch.

Collect public social media data at scale with Bright Data →

FAQ

Scraping data that any anonymous visitor can see is treated differently from accessing private data, but major platforms — Instagram, TikTok, and Facebook — prohibit automated access in their Terms of Service regardless. Laws on scraping public data also vary by jurisdiction. Get qualified legal advice before deploying a commercial scraper against any of these platforms.

Instagram, TikTok, and Facebook all block known datacenter and cloud IP ranges at the CDN or WAF layer before serving any content. Residential or ISP proxies with realistic TLS fingerprints are the minimum entry requirement for consistent access to public pages.

Public profile pages on TikTok and Facebook are heavily React-rendered; useful data (video counts, bio, post metadata) appears only after JavaScript executes. Instagram serves some data server-side via Open Graph meta tags. The Bright Data Web Unlocker handles rendering automatically, so you submit a plain HTTP request and receive the fully hydrated page.

Without a login you can access: display name, bio, follower and following counts (where shown publicly), post captions and hashtags, public engagement metrics (likes, comments), and profile images — all carried through Open Graph meta tags and, on some platforms, embedded JSON-LD or page-state objects.

What Public Social Data You Can Collect

Public Profile Pages

Post and Hashtag Feeds

Engagement Signals and Sentiment Indicators

Why Social Platforms Are Hard Scraping Targets

Prerequisites

Extracting Open Graph Metadata from Public Profiles

PHP

Node.js

Rust

Architecture for a Sentiment-Monitoring Pipeline

Legal and Ethical Considerations

Scaling to Production

FAQ

Is scraping public social media data legal?

Why do residential proxies matter for social media scraping?

Do I need a headless browser to scrape social media profiles?

What data is available on public social media profiles?