User-agent: * Disallow: / User-agent: GPTBot User-agent: AdsBot-Google Disallow: / User-agent: facebookexternalhit User-agent: FacebookExternalHit User-agent: meta-externalfetcher User-agent: Googlebot Allow: /