Syntax

  • User-agent- Following instructions are for a specific bot (or * for all)
  • Allow- Encourages bot to call following URL
  • Disallow- Robot should not crawl pages