User-agent: * Crawl-delay: 240 User-agent: Bingbot Crawl-delay: 5 User-agent: Googlebot Crawl-delay: 1 User-agent: * Allow: / User-agent: * Disallow: /sgns/ Disallow: /cu/ Disallow: /od_sim* Disallow: /od-tables Disallow: /od-adv-tables User-agent: BLEXBot User-agent: AhrefsBot User-agent: DataForSeoBot User-agent: SEOkicks User-agent: SemrushBot User-agent: DotBot User-agent: AwarioSmartBot User-agent: PetalBot User-agent: MJ12bot Disallow: / Sitemap: https://lohntastik.de/sitemaps/sitemap.xml Sitemap: https://lohntastik.de/sitemaps/gns.xml Sitemap: https://lohntastik.de/sitemaps/salary_search_sitemap.xml Sitemap: https://lohntastik.de/sitemaps/TVL-TVoeD.xml