User-agent: Yandex Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ User-agent: * Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 10 # Мусорные SEO-краулеры — отбиваем сразу. Они не дают полезного трафика, # но регулярно роняют /companies/* в 504 параллельными запросами. User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: DataForSeoBot Disallow: / # Amazonbot перенесён в AI-allowlist выше — здесь не дублируем, # иначе trash-block перебивает allowlist для одного UA. User-agent: GPTBot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: OAI-SearchBot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: ChatGPT-User Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: ClaudeBot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: Claude-Web Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: anthropic-ai Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: PerplexityBot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: Perplexity-User Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: Google-Extended Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: Bytespider Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: CCBot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: YouBot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: DuckAssistBot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: Meta-ExternalAgent Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: FacebookBot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: facebookexternalhit Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: Amazonbot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: Applebot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: Applebot-Extended Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: YandexAdditional Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: MistralAI-User Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: cohere-ai Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 User-agent: Diffbot Allow: / Allow: /arenda/ Allow: /catalog/ Allow: /news/ Allow: /blog/ Allow: /help/ Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /sitemap.xml Allow: /sitemap-landings-v2.xml Disallow: /~re* Disallow: *?sort=* Disallow: *&sort=* Disallow: /user/login Disallow: /user/registration Disallow: /user/recovery Disallow: /user/settings Disallow: /user/security Disallow: /my-orders/ Disallow: /my-ads/ Disallow: /profile/ Disallow: /cabinet/ Disallow: /messages/ Disallow: /search?qsearch=* Disallow: /poisk/*?q=* Disallow: /feed/avito/* Disallow: /feed/perevozka24/* Disallow: /admin/ Disallow: /debug/ Disallow: /api/ Crawl-delay: 1 Host: https://vsteh.ru Sitemap: https://vsteh.ru/sitemap.xml # Дополнительные карты для конкретных кластеров URL: # Sitemap: https://vsteh.ru/sitemap-landings.xml (SEO-лендинги /arenda/*) # LLM / agent discovery manifest: # https://vsteh.ru/llms.txt