Publish the file at the root
In an Astro frontend, a public/robots.txt file is served at /robots.txt. Add User-agent rules, any intended Disallow paths and an absolute Sitemap URL. Review these instructions against your actual routes before deployment.
Understand its limits
Robots.txt controls crawling for cooperating bots. It does not protect sensitive content and does not reliably remove a URL from search. Use authentication for private content and noindex for publicly accessible pages you do not want indexed.