Configure Robots.txt will only appear if the plugin is enabled.
Location: Site > Optimization > Configure Robots.txt
Here you can set up your site to prevent robots from indexing the web and direct search results of your AnswerHub site.
In the Content field, enter the Sitemap, Disallow, and User-Agent callouts as follows:
- Sitemap: Enter the location of your sitemap.xml file.
- User-Agent: Enter the specific bots to deny access to.
If there is more than one, create as many User-Agent callouts as necessary, separated by a double space.
- Under each instance of User-Agent, you will find the word Disallow.
- Disallow: Enter the directory you want to deny the specified bot access to (for example, /users/, /comments, /search, etc.)
If there is more than one, create as many Disallow callouts as necessary with each one on a separate line.
- Click the SAVE button.
A successful save will result in a green bar at the top of the page with a message that reads "Robots.txt file successfully saved."
Updated about 2 years ago