Usage

1. Add robots.php file in the theme directory.

For example:

• Theme: placeholder-theme

• Location: wp-content/themes/placeholder-theme/robots.php

The robots.php file is used to dynamically generate the robots.txt content for the theme. The file is accessed when search engine crawlers request the robots.txt file for SEO directives.

1. In the repository nginx/app/locations.conf add these lines.

location /robots.txt { 

  log_not_found off; 

  access_log off; 

  try_files $uri $uri/ /wp-content/themes/placeholder-theme/robots.php; 

  location ~ /robots.txt { 

    rewrite /robots.txt /wp-content/themes/placeholder-theme/robots.php break; 

    header_filter_by_lua_block { 

      local ttl = ngx.var.upstream_http_expires or 60 

      ngx.header["Cache-Control"] = "max-age=" .. ttl 

      ngx.header["expires"] = ttl 

      ngx.header["Content-Type"] = "text/plain; charset=utf-8" 

    } 

    fastcgi_param REMOTE_ADDR           $http_x_real_ip; 

    # ... (other fastcgi_params) 

    fastcgi_pass php-fpm; 

  } 

}  

This configuration allows each theme to manage its own robots.txt file through a dynamic robots.php script. The try_files directive checks for the existence of the file, and if not found, redirects to the specified PHP script.

Sample robots.php file:

<?php 

// Silence is golden. 

header("Content-Type: text/plain"); 

echo "User-agent: *\r\n"; 

echo "Disallow: /wp-admin/\r\n"; 

echo "\r\n"; 

echo "Sitemap: https://" . $_SERVER['HTTP_HOST'] . "/sitemap.xml"; 

echo "Sitemap: https://" . $_SERVER['HTTP_HOST'] . "/sitemap.xml"; 

Specifies the location of the XML sitemap for search engines to find and index.