@robinpath/robots

robots.txt parsing, generation, URL permission checking, and crawl configuration

Why use this module?

The robots module lets you:

• Parse robots.txt content
• Generate robots.txt
• Check if URL is allowed
• Check if URL is disallowed
• Get crawl delay for agent

All functions are callable directly from RobinPath scripts with a simple, consistent API.

Installation

npm install @robinpath/robots

Quick Start

No credentials needed — start using it right away:

robots.create {"rules": [{"userAgent": "*", "disallow": ["/admin"]}]}

Available Functions

Function	Description
robots.parse	Parse robots.txt content
robots.create	Generate robots.txt
robots.isAllowed	Check if URL is allowed
robots.isDisallowed	Check if URL is disallowed
robots.getCrawlDelay	Get crawl delay for agent
robots.getSitemaps	Get sitemap URLs
robots.getRules	Get rules for agent
robots.addRule	Add rule to robots config
robots.removeRule	Remove rule by user agent
robots.addSitemap	Add sitemap URL
robots.allowAll	Generate allow-all robots.txt
robots.disallowAll	Generate disallow-all robots.txt
robots.fetch	Fetch and parse robots.txt from URL

Examples

Generate robots.txt

robots.create {"rules": [{"userAgent": "*", "disallow": ["/admin"]}]}

Check if URL is allowed

robots.isAllowed $robots "/page" "Googlebot"

Check if URL is disallowed

robots.isDisallowed $robots "/admin"

Integration with RobinPath

import { RobinPath } from "@wiredwp/robinpath";
import Module from "@robinpath/robots";

const rp = new RobinPath();
rp.registerModule(Module.name, Module.functions);
rp.registerModuleMeta(Module.name, Module.functionMetadata);

const result = await rp.executeScript(`
  robots.create {"rules": [{"userAgent": "*", "disallow": ["/admin"]}]}
`);

Full API Reference

See MODULE.md for complete documentation including all parameters, return types, error handling, and advanced examples.

Related Modules

• @robinpath/json — JSON module for complementary functionality

License

MIT