User-agent: Mediapartners-Google Disallow: #All search engines are controlled by the lines below, which also enable all blog entries and pages while blocking all other searches. User-agent: * Disallow: /search* Disallow: /20* Allow: /*.html Allow: /ads.txt #sitemap of the blog Sitemap: https://www.example.com/sitemap.xml Sitemap: https://www.example.com/sitemap-pages.xml