How To Fix the Indexed Though Blocked by robots.txt Error
How To Fix the Indexed Though Blocked by robots.txt Error
It can be discouraging to see your website’s decline in search rankings. 📉 This post is here to help 💪
👉 Subscribe: https://www.youtube.com/c/Kinsta?sub_…
It can be discouraging to see your website’s decline in search rankings. When your pages are no longer crawled by Google, these lower rankings can contribute to fewer visitors and conversions.
The “Indexed, though blocked by robots.txt” error can signify a problem with search engine crawling on your site. When this happens, Google has indexed a page that it cannot crawl. Fortunately, you can edit your robots.txt file to specify which pages should or should not be indexed.
In this video, we’ll explain the “Indexed, though blocked by robots.txt” error and how to test your website for this problem. Then, we’ll show you two different methods of fixing it. Let’s get started!▶️
ℹ️ About Kinsta
Kinsta provides Application, Database, and Managed WordPress Hosting. Powered by Google Cloud Platform’s premium tier network and compute-optimized C2 virtual machines, Kinsta delivers enterprise-level security, a free CDN global edge network with 200+ PoPs, and 24/7/365 support.
🚀 Don’t let unnecessarily complicated and underperforming hosts delay your projects. Explore our plans and resource-based pricing to find the perfect fit to run your applications, databases, or WordPress sites ► https://kinsta.com/plans/
💡Learn more about Kinsta’s hosting solutions:
👉Application Hosting: https://kinsta.com/application-hosting/
👉Database Hosting: https://kinsta.com/database-hosting/
👉Managed WordPress Hosting: https://kinsta.com/wordpress-hosting/
👤Follow us:
► https://kinsta.com/blog/
► https://kinsta.com/newsletter/
► https://twitter.com/kinsta
► https://instagram.com/kinstahosting
► https://facebook.com/kinstahosting
🕘Timestamps
0:00 Intro
0:47 Finding the Source of the “Indexed, Though Blocked by robots.txt” Error
2:19 Edit robots.txt Directly
3:49 Use an SEO Plugin
4:14 Yoast SEO
4:54 Rank Math
6:23 Squirrly SEO
📚 Resources
🛠 Edit robots.txt Directly
► https://kinsta.com/knowledgebase/inde…
#Robotstxt #IndexedThoughBlockedByRobotstxtError
Content
0 -> Hey there, I'm Mike, and today I'm going to show you
2.31 -> how to fix the blocked by robots.txt error.
6.723 -> (upbeat music)
11.28 -> The index, though, blocked by robots.txt error
14.43 -> can signify a problem with search engines
16.32 -> crawling on your site.
17.76 -> When this happens,
18.69 -> Google has an indexed page that it cannot crawl.
21.63 -> Fortunately, you can edit your robots.txt file
24.54 -> to specify which pages should or should not be indexed.
28.35 -> In this video,
29.19 -> we'll show you two different methods of fixing it.
31.5 -> But before we get started,
32.76 -> I wanna let you know that there'll be links
34.2 -> to more resources in the video's description,
36.42 -> and remember, subscribe, and ring that bell
39.06 -> to get notifications for future helpful content.
41.55 -> Now, let's learn how to find the source of this error.
50.13 -> One effective way to identify problems
52.26 -> in page indexing is to sign into Google Search Console.
55.74 -> After you verify site ownership, you'll be able to
58.44 -> access reports about your website's performance.
61.59 -> In the index section, click on the valid with warnings tab.
65.13 -> This will pull up a list of your indexing errors
67.65 -> including any index though blocks
69.96 -> by robots.txt warnings.
72.12 -> If you don't see any
73.14 -> your website likely isn't experiencing this problem.
75.99 -> Alternatively, you can use Google's robots.txt tester.
79.62 -> With this tool
80.61 -> you can scan your robots.txt file to look
83.34 -> for syntax warnings and other errors.
86.25 -> At the bottom of the page
87.54 -> enter a specific URL to see if it's blocked.
90.45 -> You'll need to choose the user agent
92.07 -> from the dropdown menu and select test.
94.59 -> You can also navigate to your domain.com/robots.txt.
98.52 -> If you already have a robots.txt file
100.71 -> this will allow you to view it.
103.29 -> Now look for disallow statements.
105.96 -> Site administrators can add these statements to
108.15 -> instruct search crawlers
109.41 -> on how to access specific files or pages.
112.74 -> If the disallow statement blocks all search engines
115.35 -> it might look like this.
117.78 -> It may also block a specific user agent.
121.297 -> With any of these tools, you'll be able to
123.54 -> identify any issues with your pages indexing.
126.36 -> Then you'll need to take action to update your
129.03 -> robots.txt file.
135.72 -> If you have a WordPress website
138.022 -> you'll probably have a virtual robots.txt file.
140.13 -> You can visit it by searching
141.27 -> for your domain.com/robots.txt in a web browser.
145.26 -> However, this virtual file won't enable you to make edits.
148.47 -> To start editing robots.txt
150.57 -> you'll need to create a file on your server.
152.85 -> Firstly, choose a text editor and create a new file.
156.27 -> Be sure to name it robots.txt.
161.04 -> Then you'll need to connect to an SFTP client.
164.13 -> If you're using a Kinsta hosting account
166.02 -> sign into My Kinsta and navigate to sites and then info.
169.89 -> You'll find your username,
171.09 -> password host, and port number here.
173.61 -> You can then download an SFTP client such as FileZilla.
177.3 -> Enter in your SFTP login credentials
179.76 -> and click on Quick Connect.
181.8 -> Lastly, upload the robots.txt file
184.38 -> to your route directory.
185.64 -> For WordPress sites,
186.72 -> it should be called public underscore html.
189.54 -> Then you can open the file and make the necessary changes.
193.26 -> You can use allow and disallow statements to
195.42 -> customize the indexing of your WordPress site.
198.21 -> For example, you may want a certain file to be crawled
201 -> without indexing the entire folder.
203.22 -> In this case, you can add this code:
207.03 -> Remember to target the page causing the index
209.76 -> though blocked by robots.txt error,
212.07 -> during this process.
213.6 -> Depending on your goal, you can specify
215.52 -> whether Google should or should not crawl the page.
218.46 -> When you're finished, save your changes.
220.74 -> Then go back to the Google search console to see
223.65 -> if this method resolved the error.
225.18 -> If you have an SEO plugin activated, you won't
227.61 -> have to create an entirely new robots.txt file.
230.76 -> In many cases, the SEO tool will build one for you.
234.18 -> Plus it may also provide ways to edit the file
236.64 -> without leaving the WordPress dashboard.
238.41 -> One of the most popular SEO plugins is Yoast SEO.
242.04 -> It can provide a detailed
243.3 -> on page SEO analysis along with additional tools
246.48 -> to customize your search engine indexing.
248.88 -> To start editing your robots.txt file,
251.16 -> go to Yoast SEO then tools in your WordPress dashboard.
255.27 -> From the list of built-in tools, select the file editor.
258.57 -> Yoast SEO will not automatically create a robots.txt file.
263.16 -> If you don't already have one,
264.54 -> click on Create robots.txt file.
268.83 -> This will open a text editor
270.24 -> with a content of your new robots.txt file.
273.45 -> Similar to the first method you can add
275.55 -> Allow Statements to pages you want to be indexed.
278.31 -> Alternatively, use disallow statements
280.38 -> for URLs to avoid indexing.
283.037 -> After making your changes, save the file.
285.9 -> Yoast SEO will alert you when you've updated the
288.51 -> robots.txt file.
291.18 -> Rank Math is another freemium
292.86 -> plugin that includes a robots.txt editor.
296.07 -> After activating the tool on your WordPress site
298.44 -> go to Rank math then general settings,
300.78 -> and then edit robots.txt.
304.26 -> In the code editor
305.31 -> you'll see some default rules, including your site map.
308.22 -> To update its settings
309.3 -> you can paste or delete code as necessary.
312.03 -> During this editing process, make sure you use one
314.73 -> or more groups,
315.72 -> with each group containing multiple rules.
317.67 -> Start each group with a user agent
319.53 -> and follow with specific directories or files,
322.17 -> and finally, assume that any webpage allows indexing
325.65 -> unless it has a disallow rule.
327.72 -> Keep in mind that this method is only possible when
330.21 -> you don't already have a robots.txt file.
333.09 -> If you do, you'll have to edit the robots.txt file
335.91 -> directly using an SFTP client.
338.49 -> Alternatively, you can delete this preexisting file
341.25 -> and use the Rank Math editor instead.
343.47 -> Once you disallow a page in robots.txt
345.99 -> you should also add a no index directive.
348.66 -> It will keep the page private from Google searches.
350.91 -> To do this, navigate to rank math,
353.19 -> then titles in meta, and then posts.
357 -> Scroll down to post robots meta and enable it.
359.85 -> Then select no index.
362.43 -> Finally, save your changes.
364.26 -> In Google Search console, find the index though blocked
367.2 -> by robots.txt warning and click on Validate fix.
370.8 -> This will enable Google to recrawl the given URLs
373.74 -> and resolve the error.
374.91 -> Using the Squirrly SEO plugin,
376.59 -> you can similarly edit robots.txt.
379.32 -> To get started, click on Squirrly SEO
381.42 -> and then SEO configuration.
383.43 -> This will open the tweaks and site map settings.
388.08 -> On the left-hand side, select a robots file tab.
390.795 -> Then you'll see a robots.txt file editor
393.63 -> that looks similar to other SEO plug-ins.
397.02 -> Using the text editor, you can add allow,
399.18 -> or disallow statements to customize the robots.txt file.
403.29 -> Continue to add as many rules as you need.
405.51 -> When you're happy with how things look,
407.22 -> select save settings.
408.75 -> Additionally, you can add no index rules
411.18 -> to certain post types.
412.62 -> To do this, you'll simply need to disable the
414.93 -> Let Google Index setting in the automation tab.
417.99 -> By default, SEO Squirrly will leave this enabled.
423.23 -> If you're looking to boost your site's organic traffic
424.89 -> and beat out your competition
426.66 -> then we have the perfect ebook for you.
429.18 -> Download our free Ultimate Guide to SEO ebook to gain access
433.65 -> to 50 plus tips we've used to grow our own organic traffic
437.67 -> by more than 123% year after year.
441.18 -> Thank you so much for watching.
442.65 -> If you found this video helpful
444.03 -> please don't forget to like, subscribe,
446.01 -> and hit the notification bell
447.21 -> for more tutorials, explainers,
449.37 -> and helpful content like this.
451.848 -> (upbeat music)
Source: https://www.youtube.com/watch?v=Sg02WPPGO7w