The scraping fallback cheat sheet

#1. Prepare the export

Normalize domains, remove duplicates and exclude sites already in your master list. Keep the source competitor and imported metrics. Set your niche and metric thresholds before crawling; keep Semrush AS and Ahrefs DR in separate columns.

#2. Use this fallback order

1. 1

   Jina Reader

   Request the page through https://r.jina.ai/[full page URL]. Check that the response contains the page’s real content. Move on if it errors, returns a challenge or provides too little useful text.

2. 2

   BeautifulSoup

   Fetch the HTML directly and parse its title, main text and relevant links. Use this when content exists in the HTML. BeautifulSoup parses what you fetch; it does not render JavaScript or clear an access block.

3. 3

   Playwright

   Render the page in a browser, wait for the main content, then extract visible text and links. This helps with JavaScript-rendered pages. Verify that you captured the article rather than an error or challenge screen.

4. 4

   Crawl4AI

   Try browser crawling and Markdown extraction when rendered output is still messy or incomplete. It can make extraction easier, but it can encounter the same access restrictions as other browser tools.

5. 5

   Kimi WebBridge

   Inspect the public page through your connected browser as the final fallback. Capture useful text and evidence URLs. If you still cannot access the content, mark the site Manual review.

Stop at the first usable result. Log the method, URL and outcome. For a 429 response, respect the retry delay and reduce the request rate. Repeated access denial or a challenge belongs in Manual review.

#3. Qualify the site with evidence

• Niche fit

  Save a page URL showing that the site covers your target topic.

• Editorial activity

  Find an actual article, guide or maintained blog section.

• Outside contributions

  Look for an explicit contributor or submission page and save its URL. Otherwise mark acceptance Unconfirmed. A blog alone does not prove that a site takes guest posts.

• Decision

  Keep relevant sites with editorial evidence. Reject clear mismatches. Use Manual review when access or relevance remains unclear. Confirm contribution terms before outreach.

#4. Return a list you can audit

Use one row per domain. Keep these columns:

1. Domain
2. Source competitor
3. Semrush AS
4. Ahrefs DR
5. Niche evidence URL
6. Editorial evidence URL
7. Contribution status
8. Contribution evidence URL
9. Extraction method
10. Decision
11. Reason

Keep imported metrics as supplied. Leave missing values blank. Record contribution status as Explicit submissions page found or Unconfirmed; a submissions page is not a placement guarantee.

Domain,Source competitor,Semrush AS,Ahrefs DR,Niche evidence URL,Editorial evidence URL,Contribution status,Contribution evidence URL,Extraction method,Decision,Reason

#Copy-paste agent brief

Replace the bracketed fields with your niche, thresholds and existing list, then give this brief to your agent.

Before processing the CSV, verify a working Python environment; HTTP access to Jina Reader; requests + beautifulsoup4; playwright + Chromium; crawl4ai with crawl4ai-setup completed; and Kimi Browser Extension/WebBridge with its local service running and Chrome/Edge connected. Test each method on one public URL. Report Ready, Missing setup or Failed for each, with any fix needed. Set up missing tools where possible; identify any credentials or browser action I must provide. Begin only after all setup checks pass.

Once ready, qualify this CSV for [niche] using [metric thresholds]. Deduplicate domains, exclude [existing list], and preserve imported Semrush AS/Ahrefs DR. Follow the fallback order above, stopping at usable content. Record exact URLs for niche fit, editorial activity and contribution evidence. Set contribution status to Explicit submissions page found only with a supporting page; otherwise Unconfirmed. Return the columns above as CSV, using Keep, Reject or Manual review with a reason. Leave unknowns blank; do not invent metrics, evidence or acceptance.

#Before scaling up

Run a small sample first. Open the evidence URLs for every Keep, check that the content supports the decision, and review anything the agent could not verify. Then process the remaining export.

#Tool documentation

Use the official documentation when setting up or checking each method.

• Jina Reader
• BeautifulSoup
• Playwright
• Crawl4AI
• Kimi WebBridge