All docs
Webflow Sitemap

Mastering Robots.txt Configuration in Webflow

Updated on Oct 10, 2023
Share this doc
Copied !

The robots.txt file is a text file placed on the website server to instruct search engine crawlers on how to interact with the site's pages. It specifies which areas of the site should be crawled and indexed, and which should be excluded. This file helps control search engine access and ensures privacy for sensitive content.

Ensure proper search engine crawling and indexing by setting up your robots.txt file on Webflow. Control access to your website's content effectively.

Setup your robots.txt file on Webflow

  1. Go to the Project Settings
  2. Go to the SEO tab
  3. On the robots.txt area field, add the following : User-agent:* if you want all bots to crawl all your pages
  4. If you want to exclude some page, you can add Disallow: /your-page-name
  5. If you want to exclude folder, add Disallow: /folder-name*

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

Unlock Webflow features thanks to Sneaps