Cross-Platform Monitoring & Collection

What this covers

Watching things over time instead of looking once, and holding what you collect so you can analyse it later without re-collecting. Two distinct jobs with different tooling: bulk collection of a corpus for analysis, and change detection on a small number of pages you care about. The third thing, which is not optional, is doing either without the pattern of your collection becoming visible to the people you are collecting on.

Method

1. Decide what you are watching before you build anything. A corpus for network analysis and an alert on one bio edit need completely different infrastructure, and building the first when you needed the second is the usual waste.
2. Prefer the platform’s own feed. A native RSS or JSON endpoint is stable, keyless, polite and does not attribute activity to an account. Reach for a scraper only when no feed exists.
3. Store raw, analyse later. Keep the untouched response alongside anything you derive from it. You will want to re-run the analysis with a different question, and you will not want to re-collect under a worse rate limit.
4. Record gaps as gaps. A failed poll is a hole in your data, not a quiet day. Log the failures next to the results or your time series will lie to you.
5. Set the cadence to the question. Hourly is almost never justified. A daily poll that runs for a year is more valuable, and far less conspicuous, than a five-minute poll that gets you blocked in a week.
6. Baseline before you conclude. Coordination, surges and silences only mean something against what normal looks like for that topic.

The judgement calls: whether you need the content or only the fact that it changed — the second is far cheaper and far quieter. Whether the collection needs an account at all, because the moment it does, the collection has an identity and a history. And whether you are allowed to keep what you are about to collect, which is a question to settle at the start rather than after you have a database of it.

Collection hygiene

Monitoring is repeated, scheduled, patterned contact with a target’s content. One look is invisible. The same request every fifteen minutes from one address for six months is a signature, and on some platforms it is a signature attached to a logged-in account.

Account
  - never your own, and never one that shares a recovery phone or email with your own
  - a research account per investigation where the platform allows it; one burned
    account should not cost you the others
  - assume every logged-in query is retained and attributable, including search terms
  - some platforms notify a user when a profile is viewed; know which before you look

Rate
  - daily is the default. Justify anything faster to yourself in writing
  - randomise the interval. A poll at exactly :00 every hour is machine-obvious
  - respect the stated limit, and treat a 429 as a signal to back off for hours,
    not to retry in a loop
  - set a real, honest User-Agent that identifies the project. It gets you unblocked
    more often than a spoofed browser string does

Network
  - one address for all of it correlates every collection you run
  - a residential or VPN exit is a trade: less rate limiting, more attribution risk
    to whoever pays for it
  - Tor is blocked by most of these platforms, so it is rarely the answer here

Footprint
  - log every request you make, with timestamp and response code. You need it to
    distinguish "they stopped posting" from "we got blocked"
  - decide retention up front. Bulk personal data attracts obligations even when
    every item was public

The archiving and provenance side of this — hashes, snapshots, what makes a collected item citable later — is on Archiving & Evidence. Per-platform surfaces, query syntax and the specific scrapers are on Social Media Platforms; this sheet is about running those things repeatedly rather than once.

Key tools

4CAT

A self-hosted capture-and-analysis platform. It is the serious option because collection and analysis live in the same place: the dataset you capture stays on your machine, and you can re-run a different analysis over it months later without touching the platform again. That is the thing no hosted service gives you.

# the documented install is the compose file plus its .env, not a clone
mkdir 4cat && cd 4cat
curl -O https://raw.githubusercontent.com/digitalmethodsinitiative/4cat/master/docker-compose.yml
curl -O https://raw.githubusercontent.com/digitalmethodsinitiative/4cat/master/.env

# the two settings worth reading before you start it:
#   SERVER_BIND_ADDRESS=127.0.0.1   localhost only (the shipped default)
#   PUBLIC_PORT=80                  the port the web UI lands on
grep -E 'SERVER_BIND_ADDRESS|PUBLIC_PORT|DOCKER_TAG' .env

docker compose up -d
docker compose logs -f backend      # first start builds indexes; wait for it

# the one-time admin link is printed in the logs, not emailed
docker compose logs frontend | grep -i 'create a new user\|token'

Creating a dataset, once it is up:

1. http://localhost:80 -> "Create dataset".
2. Pick the data source. Natively it collects 4chan, 8kun, Bluesky, Telegram, Tumblr
   and TikTok (from a list of URLs). Everything else arrives as an upload.
3. Set the query and the date range. Date range is the field people leave open and
   then wonder why the job runs for a day -- bound it.
4. Submit, then leave it. The backend queues and processes; the dataset appears in
   your list with a status. Big Telegram or 4chan pulls take hours.
5. On the finished dataset, run processors rather than exporting immediately:
   word frequencies, co-word networks, time series, top posters, image downloads
   and clustering. Each produces a child dataset you can chain further processors on.
6. Export the one you want as CSV or NDJSON, and keep the parent dataset. The parent
   is the thing you cannot re-create.

The platform coverage is the live constraint and it changes: X/Twitter, Instagram, LinkedIn, Threads and Pinterest are no longer collected by 4CAT directly — they come in through Zeeschuimer below, or as an upload from elsewhere. Datasets are big, and the Docker volumes will fill a disk without warning, so check free space before a large pull. And a 4CAT capture is a snapshot of what the platform served at that moment: edits and deletions after the capture are invisible to it, which is a feature when you want the original and a trap when you assume it reflects the live site.

Zeeschuimer

The companion extension, and the current answer to “how do I get X, Instagram or TikTok data into 4CAT”. It watches the data the platform sends to your browser as you scroll, and keeps it. No API, no scraping requests of its own — it records what your normal browsing already fetched.

1. Firefox only. Signed .xpi builds are on the project's releases page; Chrome is
   not supported.
2. Install, then open the extension and enable the platform you are about to browse.
   Supported: TikTok, Instagram, Threads, X/Twitter, Pinterest, Gab, Truth Social,
   9gag, Imgur, Douyin and RedNote/Xiaohongshu.
3. Browse normally -- a profile, a hashtag, a search. The item counter in the
   extension ticks up as content loads. Content you never scrolled past is not
   captured, because it was never sent to your browser.
4. Export as NDJSON or CSV, or put your 4CAT URL in the extension and upload straight
   into it as a dataset.
5. Record how you browsed: which profile, which tab, how far you scrolled. That is
   the sampling frame, and without it the dataset's coverage is undocumented.

This is manual collection with automatic recording, so it does not scale and it does not run unattended — which is also why it survives platform changes better than scrapers do. It captures through your logged-in session, so everything you collect is attributable to that account: use a research profile in a separate Firefox container or profile, never your own. Platform support needs constant maintenance and individual platforms do break; check the releases page before assuming the extension is at fault rather than the site.

changedetection.io

Self-hosted page watching. This is the one to run when the thing you need to notice is an edit — a bio rewritten, a post deleted, an officer removed from a company page, a policy document quietly amended, a price or a staff list changing.

# docker, bound to localhost
docker run -d --restart always -p "127.0.0.1:5000:5000" \
  -v datastore-volume:/datastore \
  --name changedetection.io dgtlmoon/changedetection.io

# or as a package, if you would rather not run a container
pip3 install changedetection.io
changedetection.io -d /path/to/empty/data/dir -p 5000

1. http://localhost:5000 -> "Add new watch", paste the URL.
2. Open "Edit" and set the filter before you set anything else. CSS selector, XPath,
   JSONPath or jq -- point it at the narrowest element carrying the signal. Watching
   a whole page means alerting on rotating ads, view counters and timestamps.
   The "Visual Selector" tab picks the element by clicking it.
3. Set "Ignore text" for the lines you know churn: relative dates, "N views",
   cookie-banner text.
4. Recheck time: per-watch. Daily for most things. The default applies to every watch
   you add, so set it low once rather than per watch.
5. Notifications: Discord, email, Slack, Telegram or a webhook via Apprise. A webhook
   into your own notes or ticketing is the one that leaves a record.
6. For a page that renders client-side, enable the Playwright/Sockpuppetbrowser
   fetcher for that watch -- the default plain fetch sees an empty shell and will
   report "no change" forever.
7. "Browser Steps" handles a page behind a form: click, fill, submit, then diff what
   comes back.
8. Every change is stored as a snapshot with a diff view, which is the part that makes
   it evidence rather than an alert.

Running it yourself means your watch list is not a third party’s business record, which matters when the watch list itself is sensitive. The cost is that a JS-heavy watch runs a real browser and is far heavier than a text diff — a dozen of those on a small VPS will struggle. And a watch only sees what an unauthenticated fetch from your server sees: a page that needs a login, or that geo-varies, needs Browser Steps or will silently watch the wrong thing.

Distill

The hosted equivalent, for when you will not run infrastructure. Same idea, less setup, and a free tier that is genuinely usable for a handful of watches: 25 monitors total but only 5 in the cloud, a 6-hour minimum cloud interval, 1,000 cloud checks a month, 2 devices, and 30 email alerts a month. Local monitors in the browser extension are unlimited but only run while the browser is open.

1. Browser extension or distill.io. On the page you want, click the extension and
   select the region -- it generates the selector for you.
2. Choose local (runs in your browser, unlimited, only while open) or cloud (runs
   without you, capped as above). For anything that matters, cloud.
3. Set the check interval. Anything under 6 hours is a paid feature on the free tier,
   and six-hourly is adequate for almost all of this work anyway.
4. Set the condition, not just "any change" -- Distill supports text conditions, so
   "alert when the number changes" beats "alert when the page differs".
5. Alerts to email or webhook; the email allowance is the binding constraint on free.
6. Export the watch list as JSON periodically. It is the only part that is painful to
   rebuild.

Your watch list and every snapshot live on their servers, which is the trade for not running anything. For a target that could plausibly subpoena or compromise a third party, that is the wrong trade and changedetection.io is the answer instead. The free tier’s 6-hour floor also means you will miss a post that goes up and comes down inside a window, which is precisely the kind of deletion worth catching — if that is the scenario, self-host and poll faster.

yt-dlp with a download archive

For recurring capture of a channel’s output, the archive file is the whole trick: it records the ID of everything already fetched, so the next run picks up only what is new. That turns a one-off download into a monitor you can cron.

pipx install yt-dlp

# first run: establish the archive. --break-on-existing stops as soon as it meets
# something already recorded, so later runs walk only the new items
yt-dlp --download-archive archive.txt --break-on-existing --lazy-playlist \
  -o '%(upload_date)s-%(id)s.%(ext)s' \
  'https://youtube.com/@channel/videos'

# metadata-only monitoring: no video files, just the record that it existed
yt-dlp --download-archive seen.txt --break-on-existing \
  --skip-download --write-info-json --write-thumbnail \
  'https://youtube.com/@channel/videos'

# bound it by date instead, for a backfill of a known window
yt-dlp --dateafter 20260101 --datebefore 20260401 --download-archive archive.txt URL

# subtitles, which turn a channel's output into greppable text as it arrives
yt-dlp --download-archive seen.txt --break-on-existing --skip-download \
  --write-auto-subs --sub-langs en 'https://youtube.com/@channel/videos'

# throttle it. these two flags are the difference between a monitor and a nuisance
yt-dlp --sleep-requests 2 --sleep-interval 10 --max-sleep-interval 30 \
  --download-archive archive.txt URL

# several channels in one run, each stopping at its own first-seen item
yt-dlp --break-per-input --break-on-existing --download-archive archive.txt \
  -a channels.txt

# a cap, so a misconfigured run cannot pull a thousand files overnight
yt-dlp --max-downloads 50 --download-archive archive.txt URL

The archive file is state: back it up, and never delete it to “start fresh” unless you mean to re-download everything. --break-on-existing assumes the listing is newest-first, which it is for channels and playlists and is not for some search result pages — on those, drop it and let the archive do the skipping. Extractors break when platforms change their pages, so pipx upgrade yt-dlp before blaming a URL, and a cron job that has silently failed for three weeks is worse than no monitoring at all, so alert on non-zero exits. Adding --cookies-from-browser attaches a real session to every request and makes the whole monitor attributable; avoid it unless the content genuinely requires a login. Full metadata usage is on Social Media Platforms.

Native feeds, before you reach for a bridge

Several platforms still publish perfectly good feeds that nobody uses because everyone assumes RSS died. These are keyless, stable, cheap to poll and attach to no account — the best monitoring surface available, where it exists.

# YouTube channel, by channel ID (the UC... form; @handles do not work here)
curl -s 'https://www.youtube.com/feeds/videos.xml?channel_id=UCXuqSBlHAE6Xw-yeJA0Tunw'

# a subreddit's new posts
curl -s -A 'research-monitor/1.0 (contact: you@example.org)' \
  'https://www.reddit.com/r/osint/new/.rss'

# one Reddit user's activity
curl -s -A 'research-monitor/1.0 (contact: you@example.org)' \
  'https://www.reddit.com/user/someuser/.rss'

# any Mastodon account, on any instance
curl -s 'https://mastodon.social/@Gargron.rss'

# a Bluesky profile
curl -s 'https://bsky.app/profile/bsky.app/rss'

# a GitHub user's public activity, which dates account behaviour precisely
curl -s 'https://github.com/bellingcat.atom'

# pull just the timestamps, to see cadence without reading content
curl -s 'https://www.youtube.com/feeds/videos.xml?channel_id=UCXuqSBlHAE6Xw-yeJA0Tunw' \
  | grep -oE '<published>[^<]+' | sed 's/<published>//'

Reddit rate-limits these hard and will return HTTP 429 to an anonymous, default-User-Agent client within a handful of requests; a descriptive User-Agent and a gap of several seconds between calls fixes it, and Reddit’s search.rss endpoint is throttled more aggressively than the subreddit and user feeds. YouTube’s feed carries only the most recent entries, so it monitors but does not backfill. Bluesky’s profile RSS covers posts and not replies or likes — the AT Protocol endpoints on Social Media Platforms go deeper. And X/Twitter publishes nothing of this kind: snscrape has been non-functional against it since the 2023 access changes and the public Nitter instances are largely dead, so there is no quiet monitoring surface for X at all — only a logged-in session, with everything that implies.

RSSHub and RSS-Bridge

For the platforms that killed their feeds. Both are self-hosted services that scrape a site and re-emit it as RSS or Atom, so your monitor still speaks one protocol no matter how many platforms are behind it.

# RSSHub -- the larger of the two, ~1000 routes
docker run -d --name rsshub -p 1200:1200 diygod/rsshub
# or with the compose file, which adds redis caching and a browser for JS sites
wget https://raw.githubusercontent.com/DIYgod/RSSHub/master/docker-compose.yml
docker compose up -d

# routes are paths. a Telegram public channel:
curl -s 'http://localhost:1200/telegram/channel/awesomeRSSHub'
# the route catalogue lives at docs.rsshub.app/routes/

# RSS-Bridge -- fewer routes (~450) but simpler, and a web UI that builds the URL
docker create --name=rss-bridge --publish 3000:80 \
  --volume $(pwd)/config:/config rssbridge/rss-bridge
docker start rss-bridge

# bridges are query parameters rather than paths
curl -s 'http://localhost:3000/?action=display&bridge=<BridgeName>&format=Atom'

Self-host both. The public rsshub.app demo instance returns HTTP 403 to ordinary requests and is not a reliable backend for anything you depend on, and a shared public instance makes your watch list someone else’s log file either way. Expect individual routes to break: they are scrapers wearing an RSS hat, and when a platform changes its markup the route returns an empty feed rather than an error — which reads exactly like “the target stopped posting”. Check periodically that a route still returns items, and never conclude silence from an empty bridge feed without confirming against the site.

curl and jq on a schedule

For a public JSON endpoint, a few lines beat any framework. The pattern that matters is the watermark: store the timestamp of the newest item you have seen, and ask only for things after it.

# poll Arctic Shift for new Reddit posts in a subreddit since the last run.
# full Arctic Shift usage is on the social media platforms sheet; this is the
# monitoring shape of it
AS='https://arctic-shift.photon-reddit.com/api'
STATE=~/monitor/last_seen_osint
SINCE=$(cat "$STATE" 2>/dev/null || echo 0)

curl -sS --fail -G "$AS/posts/search" \
  --data-urlencode 'subreddit=osint' \
  --data-urlencode "after=$SINCE" \
  --data-urlencode 'sort=asc' \
  --data-urlencode 'limit=100' \
  -o /tmp/new.json || { echo "poll failed $(date -u +%FT%TZ)" >> ~/monitor/errors.log; exit 1; }

# advance the watermark only on a successful fetch, or a failure silently
# becomes a gap you never notice
jq -r '.data[-1].created_utc // empty' /tmp/new.json | grep . && \
  jq -r '.data[-1].created_utc' /tmp/new.json > "$STATE"

# append raw, then derive. the raw file is the thing you cannot regenerate
cat /tmp/new.json >> ~/monitor/osint-raw.ndjson
jq -r '.data[] | [.created_utc, .author, .title] | @tsv' /tmp/new.json

# a Bluesky author feed, keyless, as a second example of the same shape
curl -sS --fail 'https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed?actor=bsky.app&limit=50' \
  | jq -r '.feed[] | [.post.indexedAt, .post.record.text] | @tsv'

# crontab: daily, at a minute that is not :00, with failures mailed to you
# 37 6 * * *  /home/you/monitor/poll-osint.sh >> /home/you/monitor/run.log 2>&1

--fail is the flag that turns a silent HTML error page into a non-zero exit, and without it your monitor will happily append a Cloudflare block page to its dataset for a month. Advance the watermark only after a confirmed good response, log every failure with a timestamp, and treat the log as part of the dataset — the difference between “they went quiet” and “we were blocked” lives there and nowhere else. Note that Bluesky’s getAuthorFeed is open without a token but searchPosts on the same public host returns 403 unauthenticated, which is the sort of asymmetry worth checking per endpoint before you build a schedule around it.

Arctic Shift for Reddit history

The live successor to Pushshift, and the only practical way to get Reddit content that has since been deleted or edited. For monitoring it plays a specific role: it is the backfill that gives your forward-looking poll a baseline, and the thing you check when a post you captured disappears.

AS='https://arctic-shift.photon-reddit.com/api'

# the baseline: what this account did before you started watching it
curl -s -G "$AS/comments/search" --data-urlencode 'author=some_user' \
  --data-urlencode 'sort=asc' --data-urlencode 'limit=100' \
  | jq -r '.data[] | [.created_utc, .subreddit] | @tsv'

# posting cadence by hour of day, which is what exposes a coordinated account
curl -s -G "$AS/posts/search" --data-urlencode 'author=some_user' \
  --data-urlencode 'limit=500' --data-urlencode 'fields=created_utc' \
  | jq -r '.data[].created_utc' \
  | python3 -c 'import sys,datetime,collections; c=collections.Counter(datetime.datetime.fromtimestamp(int(l), datetime.UTC).hour for l in sys.stdin); [print(f"{h:02d} {c[h]}") for h in range(24)]'

# did a post you captured actually get removed, or did you lose it
curl -s -G "$AS/posts/ids" --data-urlencode 'ids=t3_abc123' | jq '.data[0] | {title, selftext, removed_by_category}'

Keyword search needs an accompanying author, subreddit, link_id or parent_id — bare keyword sweeps are refused, and so are very broad author or subreddit queries. The archive holds what it ingested at the time, so an edit made after ingestion is invisible and a post deleted within seconds may never have been captured at all; a record here proves the text was published, not that it is live. Bulk dumps for offline work are published separately from the API. Query syntax in full is on Social Media Platforms.

Narrative and coordination analysis

• Posting-time clustering exposes networks. Accounts that post within seconds of each other, repeatedly, are coordinated. The jq plus histogram pattern above is enough to see it.
• Identical phrasing across accounts is the strongest single indicator of copy-paste campaigns.
• Follower overlap between accounts is more telling than follower count.
• The Information Laundromat compares content and technical metadata — ad IDs, analytics tags, registration details, code structure — across sites, to find where the same text is syndicated and which sites share infrastructure. Built by the Alliance for Securing Democracy, which merged into the Institute for Strategic Dialogue on 1 January 2026; the tool remains live at the address above. The older hyphenated domain no longer resolves.
• Graphing and clustering what you have collected is on Data Analysis & Visualisation.

Tool reference

Pitfalls

• Dead tools still appear in tutorials. snscrape and the public Nitter instances do not work against X. Bing’s Visual Search API and the rest of the Bing Search family were decommissioned in August 2025. Check a tool is alive before you design a collection around it.
• Rate limits end collections mid-run. Check for gaps before you analyse; a missing day looks like silence rather than a failure.
• An empty feed is ambiguous. A broken RSSHub route, a changed selector and a target who stopped posting all look identical downstream. Monitor your monitors.
• Sampling bias reads as a finding. If a tool only reaches accounts above some follower count, or only what you happened to scroll past, its “network” is an artefact of that cutoff.
• Coordination needs a baseline. Fans of the same thing post about it at the same time. Compare against normal behaviour for the topic before calling it inauthentic.
• Monitoring is contact. Scheduled requests are a pattern; a logged-in monitor is an attributable pattern. Decide what that costs before you start, not after.
• Storage and legality. Bulk personal data attracts data-protection obligations even when every individual item was public, and a dataset is harder to delete than to collect.

Worked example

One datum: a single Telegram channel name, from a screenshot, alleged to be seeding a story that later appeared on a cluster of news-like websites.

1. Baseline before watching. Pull the channel’s existing history into 4CAT as a Telegram dataset with a bounded date range. You now know its normal posting cadence and vocabulary, which is what any later claim of a surge has to be measured against.
2. Find the forward-looking surface. Telegram has no public feed, so an RSSHub route (/telegram/channel/<name> on your own instance) gives you a pollable endpoint. Verify it returns items today, so that an empty feed next month means something.
3. Schedule it honestly. A daily curl with a stored watermark, a descriptive User-Agent, and a failure log. Not hourly — a story that takes days to syndicate does not need fifteen-minute resolution, and the slower poll survives longer.
4. Watch the downstream sites for edits, not just posts. Each suspected site gets a changedetection.io watch filtered to the article body, so a quietly amended paragraph or a removed byline raises an alert with a stored diff. This is the part that a collection-only approach misses entirely.
5. Capture the video claims properly. The channel posts clips; yt-dlp --download-archive with --write-info-json on its linked channels picks up only what is new each day and records upload_date for each, giving every clip a latest-possible date.
6. Test the syndication claim. Feed one article URL to the Information Laundromat. Content similarity tells you the text is shared; the technical indicators — a common analytics ID across four of the sites — tell you something stronger, because wording can be copied by anyone and a shared tracking ID usually cannot.
7. Cross-check the Reddit leg. Arctic Shift for posts linking those domains, grouped by subreddit and by hour, shows whether the amplification is a handful of accounts on a schedule or genuine spread.
8. Archive as you go. Every page you will cite goes to a snapshot at the time you saw it, per Archiving & Evidence — these sites edit and disappear, which is the behaviour you are documenting.

What you can assert: a dated posting history for the channel, dated first appearances on each downstream site, stored diffs of any subsequent edits, and a shared technical indicator linking some of those sites.

What would falsify it: an RSSHub route that broke silently mid-period, which would turn a real gap into an apparent one — so the run log, showing a successful fetch every day, is doing as much evidential work as the data. A shared analytics ID that turns out to belong to a common CMS template, or to an agency that serves unrelated clients, would break the infrastructure link; check what else carries that ID before leaning on it.

Broader catalogues

• Social Media OSINT

Sources

Both catalogues below are maintained by other people and are considerably larger than this page. Use them as the canonical index; this sheet is a working route through them.

• Bellingcat’s Online Investigation Toolkit — ~340 tools, each with its own review page covering cost, difficulty, requirements and limitations.
• OSINT Newsletter Tools Library — ~280 tools, organised by investigative goal.

Neither publishes a licence, so nothing here is copied from them: tool names, one-line descriptions, cost flags and links are catalogue facts, and the method and commentary are this site’s own. See credits.