Guide

The builder tracks a current context. Until you call userAgent(), every rule you add is global and renders under User-agent: *:

use Fkrzski\RobotsTxt\RobotsTxt;

$robots = new RobotsTxt();

echo $robots
    ->disallow('/admin')
    ->allow('/public')
    ->toString();

User-agent: *
Disallow: /admin
Allow: /public

Calling userAgent() switches the context: every rule after it belongs to that crawler until you switch again.

use Fkrzski\RobotsTxt\Enums\CrawlerEnum;

echo $robots
    ->disallow('/admin')              // global
    ->userAgent(CrawlerEnum::GOOGLE)
    ->disallow('/google-only')        // Googlebot
    ->userAgent(CrawlerEnum::BING)
    ->disallow('/bing-only')          // Bingbot
    ->toString();

User-agent: *
Disallow: /admin

User-agent: Googlebot
Disallow: /google-only

User-agent: Bingbot
Disallow: /bing-only

userAgent() sets the context and leaves it set — every following rule sticks to that crawler. forUserAgent() scopes a group of rules to a crawler inside a closure, then restores the previous context when the closure returns.

The same policy, written both ways:

$robots
    ->disallow('/admin')
    ->userAgent(CrawlerEnum::GOOGLE)
    ->disallow('/private')
    ->crawlDelay(10);

$robots
    ->disallow('/admin')
    ->forUserAgent(CrawlerEnum::GOOGLE, function (RobotsTxt $robots): void {
        $robots
            ->disallow('/private')
            ->crawlDelay(10);
    });

Both render the same file:

User-agent: *
Disallow: /admin

User-agent: Googlebot
Disallow: /private
Crawl-delay: 10

The difference shows up on the next call. Append allow('/public') to each chain and they diverge:

$robots
    ->disallow('/admin')
    ->userAgent(CrawlerEnum::GOOGLE)
    ->disallow('/private')
    ->crawlDelay(10)
    ->allow('/public');   // still Googlebot — the context stayed set

User-agent: *
Disallow: /admin

User-agent: Googlebot
Disallow: /private
Crawl-delay: 10
Allow: /public

$robots
    ->disallow('/admin')
    ->forUserAgent(CrawlerEnum::GOOGLE, function (RobotsTxt $robots): void {
        $robots
            ->disallow('/private')
            ->crawlDelay(10);
    })
    ->allow('/public');   // back to global — the context was restored

User-agent: *
Disallow: /admin
Allow: /public

User-agent: Googlebot
Disallow: /private
Crawl-delay: 10

toString() always emits sections in a fixed order, regardless of the order you called the methods:

1. Global rules (under User-agent: *), if any
2. Each crawler block, in the order its crawler was first referenced
3. All sitemaps, last

Blocks are separated by a blank line. Note the two sitemaps added at opposite ends of the chain:

$robots = new RobotsTxt();

echo $robots
    ->disallow('/admin')
    ->sitemap('https://example.com/sitemap1.xml')
    ->forUserAgent(CrawlerEnum::GOOGLE, function (RobotsTxt $robots): void {
        $robots
            ->disallow('/google-private')
            ->allow('/public/*');
    })
    ->forUserAgent(CrawlerEnum::BING, function (RobotsTxt $robots): void {
        $robots
            ->disallow('/bing-private')
            ->crawlDelay(5);
    })
    ->sitemap('https://example.com/sitemap2.xml')
    ->toString();

User-agent: *
Disallow: /admin

User-agent: Googlebot
Disallow: /google-private
Allow: /public/*

User-agent: Bingbot
Disallow: /bing-private
Crawl-delay: 5

Sitemap: https://example.com/sitemap1.xml
Sitemap: https://example.com/sitemap2.xml

toFile() renders the content and writes it to a file. With no argument it writes robots.txt to the current working directory:

$robots->toFile();                           // ./robots.txt
$robots->toFile('/var/www/html/robots.txt'); // custom path

It returns true on success and throws a RuntimeException when the target directory does not exist, or the directory or an existing file is not writable.

The file has to land in the document root, so the path depends on your stack:

$robots->toFile(public_path('robots.txt'));

$robots->toFile($this->getParameter('kernel.project_dir').'/public/robots.txt');

$robots->toFile(__DIR__.'/public/robots.txt');

The policy can also be rendered per request — useful when it depends on the environment:

use Fkrzski\RobotsTxt\RobotsTxt;

Route::get('/robots.txt', function () {
    $robots = new RobotsTxt();

    if (app()->isProduction()) {
        $robots->disallow('/admin')->sitemap(url('/sitemap.xml'));
    } else {
        $robots->disallowAll();
    }

    return response($robots->toString(), 200, ['Content-Type' => 'text/plain']);
});

use Fkrzski\RobotsTxt\RobotsTxt;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Attribute\Route;

final class RobotsController
{
    #[Route('/robots.txt', name: 'robots')]
    public function __invoke(): Response
    {
        $robots = (new RobotsTxt())
            ->disallow('/admin')
            ->sitemap('https://example.com/sitemap.xml');

        return new Response($robots->toString(), 200, ['Content-Type' => 'text/plain']);
    }
}

<?php

use Fkrzski\RobotsTxt\RobotsTxt;

require __DIR__.'/../vendor/autoload.php';

header('Content-Type: text/plain');

echo (new RobotsTxt())
    ->disallow('/admin')
    ->sitemap('https://example.com/sitemap.xml')
    ->toString();

The builder does no I/O beyond toFile(), so there is nothing to fake — assert on the rendered string:

it('blocks the admin panel', function (): void {
    $robots = (new RobotsTxt())->disallow('/admin')->allow('/public');

    expect($robots->toString())->toBe(<<<'TXT'
        User-agent: *
        Disallow: /admin
        Allow: /public
        TXT);
});

public function test_it_blocks_the_admin_panel(): void
{
    $robots = (new RobotsTxt())->disallow('/admin')->allow('/public');

    $this->assertSame(<<<'TXT'
        User-agent: *
        Disallow: /admin
        Allow: /public
        TXT, $robots->toString());
}

Covering toFile() means handing it a path. Without one it resolves to getcwd() — under a test runner that is the project root, so the suite drops a stray robots.txt there, or overwrites a real one:

$path = sys_get_temp_dir().'/robots-'.uniqid().'.txt';

$robots->toFile($path); // never a bare toFile() in a test