.Google's John Mueller answered a question regarding why Google indexes web pages that are actually refused coming from crawling by robots.txt and why the it's safe to dismiss the relevant Search Console files concerning those creeps.Robot Web Traffic To Inquiry Guideline URLs.The person inquiring the concern documented that robots were actually developing links to non-existent concern parameter Links (? q= xyz) to webpages with noindex meta tags that are actually also shut out in robots.txt. What urged the inquiry is actually that Google is actually creeping the links to those webpages, getting shut out through robots.txt (without watching a noindex robots meta tag) after that acquiring turned up in Google.com Browse Console as "Indexed, though blocked out by robots.txt.".The individual asked the observing inquiry:." However below is actually the major question: why will Google index pages when they can't also see the content? What is actually the advantage in that?".Google's John Mueller verified that if they can not creep the web page they can not view the noindex meta tag. He also makes an interesting acknowledgment of the website: hunt driver, urging to ignore the end results because the "ordinary" customers will not find those outcomes.He created:." Yes, you're appropriate: if we can't creep the page, our company can not find the noindex. That stated, if our company can not crawl the webpages, at that point there's certainly not a great deal for our team to mark. Thus while you may view some of those webpages with a targeted web site:- concern, the average user won't find all of them, so I would not fuss over it. Noindex is actually likewise alright (without robots.txt disallow), it simply indicates the Links are going to end up being crawled (and also find yourself in the Explore Console document for crawled/not listed-- neither of these conditions result in issues to the remainder of the web site). The fundamental part is actually that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the restrictions being used the Internet site: hunt evolved search operator for analysis main reasons. Among those causes is actually due to the fact that it is actually certainly not connected to the routine hunt index, it is actually a distinct trait altogether.Google's John Mueller discussed the web site search operator in 2021:." The quick answer is actually that a site: query is not implied to be total, neither used for diagnostics purposes.A site concern is actually a certain type of hunt that limits the results to a specific internet site. It is actually essentially only words site, a bowel, and then the website's domain name.This inquiry confines the outcomes to a specific web site. It is actually not meant to become an extensive assortment of all the webpages from that internet site.".2. Noindex tag without utilizing a robots.txt is alright for these kinds of circumstances where a robot is actually linking to non-existent webpages that are getting uncovered by Googlebot.3. URLs with the noindex tag will generate a "crawled/not catalogued" item in Look Console which those will not have an adverse result on the remainder of the website.Review the inquiry and also respond to on LinkedIn:.Why will Google index pages when they can't even view the material?Featured Photo by Shutterstock/Krakenimages. com.