{"id":2422,"date":"2026-05-01T22:49:57","date_gmt":"2026-05-01T22:49:57","guid":{"rendered":"https:\/\/wpseonext.com\/docs\/sitemaps-robots-url-rules\/"},"modified":"2026-10-14T14:03:19","modified_gmt":"2026-10-14T14:03:19","slug":"sitemaps-robots-url-rules","status":"publish","type":"page","link":"https:\/\/coocadev.com\/seo-suite\/docs\/sitemaps-robots-url-rules\/","title":{"rendered":"Sitemaps, Robots, and URL Rules"},"content":{"rendered":"<div class=\"wpsn-structure-guide\">\n<header>\n<p class=\"wpsn-doc-eyebrow\">SEO Suite guide<\/p>\n<h2>Check discovery, indexing, and URLs separately<\/h2>\n<p>A sitemap lists URLs, robots.txt controls crawling, and indexing directives influence whether a page can appear in search. They are not interchangeable switches.<\/p>\n<\/header>\n<section class=\"wpsn-structure-section\" id=\"guide-step-1\">\n<h2>Inspect before editing<\/h2>\n<ol>\n<li>Open the sitemap URL shown by the active SEO tool and confirm that it loads.<\/li>\n<li>Check one important public page in the list and open its URL.<\/li>\n<li>Review WordPress search-engine visibility and the page&#8217;s indexing settings.<\/li>\n<li>Check robots.txt for rules that unintentionally block public content.<\/li>\n<li>Record current values before changing a rule. Test broad rules on staging first.<\/li>\n<\/ol>\n<\/section>\n<section class=\"wpsn-structure-section\" id=\"guide-step-2\">\n<h2>Example: a missing service page<\/h2>\n<p>Confirm the page is published and intended to be public. Then check whether its content type is included in the sitemap and whether it is marked noindex. Do not start by deleting all robots rules: the missing page may have a local setting or a stale cached sitemap.<\/p>\n<\/section>\n<section class=\"wpsn-structure-section\" id=\"guide-step-3\">\n<h2>Important limits<\/h2>\n<p>A sitemap does not guarantee indexing. Blocking a URL in robots.txt does not reliably remove it from search. For Google to process a page&#8217;s noindex directive, it must be able to crawl that page. See <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/block-indexing\">Google&#8217;s noindex guidance<\/a>.<\/p>\n<p>Do not use these settings as privacy protection. Private content needs access controls, not merely a search directive.<\/p>\n<\/section>\n<section class=\"wpsn-structure-section\" id=\"guide-step-4\">\n<h2>Verify the result<\/h2>\n<p>Reopen the sitemap and affected page after clearing relevant caches. Confirm that the intended public URL works and the saved directive is actually present. Search-engine processing is separate and may take time.<\/p>\n<p>Read <a href=\"https:\/\/coocadev.com\/seo-suite\/docs\/sitemaps-and-indexing\/\">Sitemaps and Indexing<\/a> for the concepts, or <a href=\"https:\/\/coocadev.com\/seo-suite\/docs\/redirects\/\">Redirects<\/a> when a URL has moved.<\/p>\n<\/section>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>SEO Suite guide Check discovery, indexing, and URLs separately A sitemap lists URLs, robots.txt controls crawling, and indexing directives influence whether a page can appear in search. They are not interchangeable switches. Inspect before editing Open the sitemap URL shown by the active SEO tool and confirm that it loads. Check one important public page [&hellip;]<\/p>\n","protected":false},"author":0,"featured_media":0,"parent":2348,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"_wpsn_focus_keyword":"","_wpsn_seo_title":"","_wpsn_meta_description":"","_wpsn_canonical":"","_wpsn_robots_index":"","_wpsn_robots_follow":"","_wpsn_og_title":"","_wpsn_og_description":"","_wpsn_og_image":"","_wpsn_twitter_title":"","_wpsn_twitter_description":"","_wpsn_twitter_image":"","_wpsn_schema_type":"","_wpsn_schema_custom_json_ld":"","_wpsn_video_url":"","_wpsn_video_title":"","_wpsn_video_description":"","_wpsn_video_thumbnail":"","_wpsn_video_duration":0,"footnotes":""},"class_list":["post-2422","page","type-page","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/coocadev.com\/seo-suite\/wp-json\/wp\/v2\/pages\/2422","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/coocadev.com\/seo-suite\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/coocadev.com\/seo-suite\/wp-json\/wp\/v2\/types\/page"}],"replies":[{"embeddable":true,"href":"https:\/\/coocadev.com\/seo-suite\/wp-json\/wp\/v2\/comments?post=2422"}],"version-history":[{"count":4,"href":"https:\/\/coocadev.com\/seo-suite\/wp-json\/wp\/v2\/pages\/2422\/revisions"}],"predecessor-version":[{"id":2829,"href":"https:\/\/coocadev.com\/seo-suite\/wp-json\/wp\/v2\/pages\/2422\/revisions\/2829"}],"up":[{"embeddable":true,"href":"https:\/\/coocadev.com\/seo-suite\/wp-json\/wp\/v2\/pages\/2348"}],"wp:attachment":[{"href":"https:\/\/coocadev.com\/seo-suite\/wp-json\/wp\/v2\/media?parent=2422"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}