Web Crawling / Merchant Website Intelligence

Module 2 · Website Intelligence

Web Crawling / Merchant Website Intelligence

Crawl and analyze merchant-owned websites to understand business processes, products/services and site history, and to detect high-risk or prohibited activities.

← Modules

RFI module deep-dive

Web Crawling / Merchant Website Intelligence

Prototype page exposes input contracts, processing stages, structured outputs, quality controls, privacy controls, integration points, PoC questions and evidence needed before production acceptance.

Open PoC Benchmark Lab

Inputs

Website URLMerchant-provided URL; normalized and domain-resolved

Declared LOBMerchant onboarding declaration

Prohibited taxonomyConfigurable risk-category policy

Source policyRobots/ToS/access method allowlist

Processing Pipeline

Policy gate

Validate source allowlist, access method, robots/ToS policy and rate-limit configuration.

Fetch & normalize

Retrieve permitted content, canonicalize URLs and deduplicate pages.

Content understanding

Classify business, product/service categories, legal/contact pages and transaction readiness.

Domain intelligence

Resolve age/WHOIS/reputation/history signals through approved connectors.

Risk-pattern detection

Identify parking, redirect chains, scam cues, prohibited terms/images and profile mismatch.

Explainable output

Return matched pages/snippets, taxonomy labels, confidence and reason codes.

Structured Outputs

detected_lobPredicted business categories + confidence

declared_matchConsistency / mismatch reason

prohibited_signalsCategory, page URL, evidence snippet

domain_historyAge / change / reputation signals

site_healthActive, parking, redirect, broken-link indicators

evidenceSource pages, timestamps and retrieval policy

✓

Website content analysisExtract business category, products/services, contact information and displayed legal information.

✓

Declared-vs-detected business classificationCompare merchant-declared LOB against website evidence.

✓

High-risk / prohibited categoriesDetect configurable categories such as unlicensed gambling, unlicensed crypto, adult content, weapons or illegal products.

✓

Domain historyEvaluate domain age, WHOIS/registration signals, changes and reputation/blacklist indicators where lawfully available.

✓

Inactive / parking / scam signalsDetect parked domains, inactive pages, suspicious redirects, impersonation and other risk indicators.

✓

Bahasa Indonesia NLPSupport Indonesian language and mixed local content.

✓

Source-policy complianceApply robots.txt, site ToS and approved-crawling policy before collection.

#	Question to provider	Prototype status	Evidence / response expected
1	How are business categories and prohibited products classified, and can categories be configured?	PoC response	Show taxonomy, rules/ML split, threshold and policy administration.
2	Which sources are used for domain history, such as WHOIS and archived web data?	PoC response	Provide data-source register and legal/contractual basis.
3	What classification accuracy is achieved and is Bahasa Indonesia supported?	PoC response	Provide Indonesian benchmark results.
4	How is robots.txt, site ToS and PDP compliance enforced during crawling?	PoC response	Provide technical policy gate and audit proof.
5	What is the output format and how does it integrate with the decision engine?	PoC response	Provide JSON schema, reason codes and evidence references.

Public ≠ unrestricted

Publicly accessible website data is still processed under a defined purpose, source policy and minimization rule.

Evidence minimization

Store only evidence required for the decision, not a full mirror of the merchant website.

Terms / robots governance

Crawling executes only after connector-specific policy approval and logs the access method.

Personal-data suppression

Personal details unrelated to merchant risk are masked or excluded from analyst views.

Illustrative structured output

Schema is intentionally explicit to support decision-engine integration, explainability and audit. Values are simulated.

{ "module":"website_intelligence", "declared_lob":"Digital Services", "detected_lob":[{"label":"Digital Marketplace","confidence":0.91}], "lob_match":0.62, "prohibited_categories":[], "domain":{"age_days":1534,"parking":false,"reputation_flags":0}, "site_readiness":{"contact":true,"terms":true,"refund":true,"cart":true}, "reason_codes":["LOB_BROADER_THAN_DECLARED"] }

Digital Footprint Investigation

Governed crawl progress, discovered pages, LOB evidence, domain history and source identity.

Open Digital Footprint →