API reference

Every public method of RobotsTxt, with its signature, behavior, and the exact output it produces. All methods return the builder for chaining, except toFile() (returns bool) and toString() (returns string).

allow(string $path): self

Adds an Allow rule for $path to the current context.

use Fkrzski\RobotsTxt\RobotsTxt;

echo (new RobotsTxt())->allow('/public')->toString();

User-agent: *
Allow: /public

disallow(string $path): self

Adds a Disallow rule for $path to the current context. Same path rules as allow() (see Path validation).

echo (new RobotsTxt())->disallow('/private')->toString();

User-agent: *
Disallow: /private

crawlDelay(int $seconds): self

Adds a Crawl-delay rule to the current context. $seconds must be non-negative.

echo (new RobotsTxt())->crawlDelay(10)->toString();

User-agent: *
Crawl-delay: 10

userAgent(CrawlerEnum $crawler): self

Switches the current context to $crawler. Every rule added afterwards belongs to that crawler until the context changes. Referencing a crawler for the first time opens its block; referencing it again reuses the same block.

use Fkrzski\RobotsTxt\Enums\CrawlerEnum;

echo (new RobotsTxt())
    ->userAgent(CrawlerEnum::GOOGLE)
    ->disallow('/private')
    ->allow('/public')
    ->toString();

User-agent: Googlebot
Disallow: /private
Allow: /public

forUserAgent(CrawlerEnum $crawler, Closure(RobotsTxt): void $rules): self

Runs $rules with the context set to $crawler, then restores the previous context. Use it to group a crawler’s rules without affecting later calls.

$robots = new RobotsTxt();

$robots->forUserAgent(CrawlerEnum::BING, function (RobotsTxt $robots): void {
    $robots
        ->disallow('/bing-private')
        ->crawlDelay(10);
});

echo $robots->toString();

User-agent: Bingbot
Disallow: /bing-private
Crawl-delay: 10

sitemap(string $url): self

Adds a sitemap. Sitemaps are context-independent — they always render together in the trailing section, in insertion order (see Sitemap validation).

echo (new RobotsTxt())
    ->sitemap('https://example.com/sitemap.xml')
    ->toString();

Sitemap: https://example.com/sitemap.xml

disallowAll(bool $disallow = true): self

Blocks the entire current context. When $disallow is true (the default) it clears the existing rules of the current context and adds a single Disallow: /. Global rules for other crawlers and all sitemaps are kept. When $disallow is false the method does nothing.

Applied globally, it wipes the previously chained rules:

echo (new RobotsTxt())
    ->allow('/public')   // cleared
    ->disallow('/admin') // cleared
    ->disallowAll()
    ->toString();

User-agent: *
Disallow: /

Applied inside a crawler context, it only clears that crawler — global rules and other crawlers survive:

echo (new RobotsTxt())
    ->disallow('/admin')                  // global — kept
    ->userAgent(CrawlerEnum::GOOGLE)
    ->allow('/public')                    // Google — cleared
    ->disallow('/private')                // Google — cleared
    ->disallowAll()
    ->userAgent(CrawlerEnum::BING)
    ->disallow('/secret')                 // Bing — kept
    ->toString();

User-agent: *
Disallow: /admin

User-agent: Googlebot
Disallow: /

User-agent: Bingbot
Disallow: /secret

toString(): string

Renders the complete robots.txt content: global block, then each crawler block, then all sitemaps, separated by blank lines. Returns an empty string when no rules have been added.

toFile(?string $path = null): bool

Writes the rendered content to a file and returns true on success. With no argument it writes robots.txt to the current working directory.

$robots = (new RobotsTxt())->disallow('/admin')->allow('/public');

$robots->toFile();                           // ./robots.txt
$robots->toFile('/var/www/html/robots.txt'); // custom path

Throws a RuntimeException when the target directory does not exist, or the directory or an existing file is not writable.

Paths pass wildcard characters through unchanged, so the standard * (match any sequence) and $ (match end of URL) patterns work as-is:

echo (new RobotsTxt())
    ->disallow('/*.php')     // block all PHP files
    ->allow('/public/*')     // allow everything under /public
    ->disallow('/private/$') // exact match for /private/
    ->toString();

User-agent: *
Disallow: /*.php
Allow: /public/*
Disallow: /private/$

Values are validated the moment you add them, so an invalid robots.txt can never be built. Bad paths, sitemaps, and crawl delays throw an InvalidArgumentException; file errors throw a RuntimeException.

Applies to allow() and disallow(). A path must:

Applies to sitemap(). A URL must:

Applies to crawlDelay(). The value must be non-negative, otherwise it throws Crawl delay cannot be negative.

use Fkrzski\RobotsTxt\RobotsTxt;

(new RobotsTxt())->disallow('admin');                    // Path must start with forward slash (/)
(new RobotsTxt())->sitemap('/sitemap.xml');              // Invalid sitemap URL format
(new RobotsTxt())->sitemap('ftp://example.com/map.xml'); // Sitemap URL must use HTTP(S) protocol
(new RobotsTxt())->crawlDelay(-1);                       // Crawl delay cannot be negative