1.6.2 • Published 1 month ago

gatsby-plugin-robots-txt v1.6.2

Weekly downloads
84,879
License
MIT
Repository
github
Last release
1 month ago

NPM version Actions Build Status AppVeyor build status Reliability Rating Coverage FOSSA Status

gatsby-plugin-robots-txt

Create robots.txt for your Gatsby site.

Install

yarn add gatsby-plugin-robots-txt

or

npm install --save gatsby-plugin-robots-txt

How To Use

gatsby-config.js

module.exports = {
  siteMetadata: {
    siteUrl: 'https://www.example.com'
  },
  plugins: ['gatsby-plugin-robots-txt']
};

Options

This plugin uses generate-robotstxt to generate content of robots.txt and it has the following options:

NameTypeDefaultDescription
hostString${siteMetadata.siteUrl}Host of your site
sitemapString / String[]${siteMetadata.siteUrl}/sitemap.xmlPath(s) to sitemap.xml
policyPolicy[][]List of Policy rules
configFileStringundefinedPath to external config file
outputString/robots.txtPath where to create the robots.txt

gatsby-config.js

module.exports = {
  plugins: [
    {
      resolve: 'gatsby-plugin-robots-txt',
      options: {
        host: 'https://www.example.com',
        sitemap: 'https://www.example.com/sitemap.xml',
        policy: [{ userAgent: '*', allow: '/' }]
      }
    }
  ]
};

env-option

gatsby-config.js

module.exports = {
  plugins: [
    {
      resolve: 'gatsby-plugin-robots-txt',
      options: {
        host: 'https://www.example.com',
        sitemap: 'https://www.example.com/sitemap.xml',
        env: {
          development: {
            policy: [{ userAgent: '*', disallow: ['/'] }]
          },
          production: {
            policy: [{ userAgent: '*', allow: '/' }]
          }
        }
      }
    }
  ]
};

The env key will be taken from process.env.GATSBY_ACTIVE_ENV first (see Gatsby Environment Variables for more information on this variable), falling back to process.env.NODE_ENV. When this is not available then it defaults to development.

You can resolve the env key by using resolveEnv function:

gatsby-config.js

module.exports = {
  plugins: [
    {
      resolve: 'gatsby-plugin-robots-txt',
      options: {
        host: 'https://www.example.com',
        sitemap: 'https://www.example.com/sitemap.xml',
        resolveEnv: () => process.env.GATSBY_ENV,
        env: {
          development: {
            policy: [{ userAgent: '*', disallow: ['/'] }]
          },
          production: {
            policy: [{ userAgent: '*', allow: '/' }]
          }
        }
      }
    }
  ]
};

configFile-option

You can use the configFile option to set specific external configuration:

gatsby-config.js

module.exports = {
  plugins: [
    {
      resolve: 'gatsby-plugin-robots-txt',
      options: {
        configFile: 'robots-txt.config.js'
      }
    }
  ]
};

robots-txt.config.js

module.exports = {
  host: 'https://www.example.com',
  sitemap: 'https://www.example.com/sitemap.xml',
  policy: [{ userAgent: '*' }]
};

Netlify

If you would like to disable crawlers for deploy-previews you can use the following snippet:

gatsby-config.js

const {
  NODE_ENV,
  URL: NETLIFY_SITE_URL = 'https://www.example.com',
  DEPLOY_PRIME_URL: NETLIFY_DEPLOY_URL = NETLIFY_SITE_URL,
  CONTEXT: NETLIFY_ENV = NODE_ENV
} = process.env;
const isNetlifyProduction = NETLIFY_ENV === 'production';
const siteUrl = isNetlifyProduction ? NETLIFY_SITE_URL : NETLIFY_DEPLOY_URL;

module.exports = {
  siteMetadata: {
    siteUrl
  },
  plugins: [
    {
      resolve: 'gatsby-plugin-robots-txt',
      options: {
        resolveEnv: () => NETLIFY_ENV,
        env: {
          production: {
            policy: [{ userAgent: '*' }]
          },
          'branch-deploy': {
            policy: [{ userAgent: '*', disallow: ['/'] }],
            sitemap: null,
            host: null
          },
          'deploy-preview': {
            policy: [{ userAgent: '*', disallow: ['/'] }],
            sitemap: null,
            host: null
          }
        }
      }
    }
  ]
};

query-option

By default the site URL will come from the Gatsby node site.siteMeta.siteUrl. Like in Gatsby's sitemap plugin an optional GraphQL query can be used to provide a different value from another data source as long as it returns the same shape:

gatsby-config.js

module.exports = {
  plugins: [
    {
      resolve: 'gatsby-plugin-robots-txt',
      options: {
        query: `{
          site: MyCustomDataSource {
            siteMetadata {
              siteUrl
            }
          }
        }`
      }
    }
  ]
};

License

FOSSA Status

gatsby-theme-sanity-evelan@cangir/gatsby-theme-phoenixsomoy-portfolio@dewen_li/gatsby-theme-bodilessgatsby-theme-kuworking-landing-onegatsby-theme-memoriestorichegatsby-theme-factlytheme-gatsby-shopifyfriday.de@afuh/gatsby-theme-up42@up42/gatsby-theme-up42@garytee/gatsby-woo-elementorgatsby-woocommerce-elementor-theme@leptest/gatsby-netlify-cmstest-theme-shawacademytest-gatsby-theme-grandesmooth-doc@wesleylhandy/gatsby-theme-author-basedaudr-blog@afuh-test/gatsby-theme-minimal@afuh/gatsby-theme-minimal@aaronconway7/create-gatsby-appgatsby-all-pack-theme-startergatsby-theme-keplergatsby-theme-kuworking-affiliategatsby-theme-kuworking-coregatsby-theme-kuworking-landing-twogatsby-theme-mdx-suite-basegatsby-theme-julesgatsby-theme-hyperobjekt-coregatsby-theme-webhart-basegatsby-theme-wvumedicinegatsby-theme-catalyst-coregatsby-theme-buzzinggatsby-theme-bootstrapgatsby-theme-portfolio-minimalgatsby-theme-octahedroidgatsby-theme-mosmangatsby-theme-municipiogatsby-theme-narrabeengatsby-theme-q3gatsby-theme-randwickgatsby-theme-rayriffy-bloggatsby-theme-sputnikgatsby-theme-stoutlabs-configsgatsby-theme-datocmsgatsby-theme-core-gwanggatsby-theme-core-testgatsby-woocommerce-themegatsby-theme-esca-boilerplategatsby-starter-zoomkodinggatsby-starter-datocmsgatsby-starter-essentialsgatsby-starter-heydaysgatsby-starter-hoodiegatsby-theme-awesomeness@laradevitt/gatsby-theme-just-basics@arshad/gatsby-theme-phoenix@kuworking/gatsby-theme-kuworking-core@mhjadav/gatsby-theme-factlyjonasbroms-www@bodiless/gatsby-theme-bodiless@brikl/gatsby-theme-brikl-store@lmack/core@cangir/gatsby-theme-casper-v3@cangir/gatsby-theme-tabler@campj/core@opensourcerepos/gatsby-theme-opensourcerepos@opensourcerepos/opensourcerepos-theme@changeinc/webapp@chr.fritsch/gatsby-theme-blog@cinemataztic/gatsby-theme-cinemataztic-cms@newrelic/gatsby-theme-newrelic@baradm100/gatsby-theme-admonation@benrobertson/gatsby-theme-seo@novvum/gatsby-theme-wiki@daniel.husar/gatsby-theme-spring@corex/gatsby-final-step@daugsbi/gatsby-theme-seo@hpprc/gatsby-theme-blog@hpprc/gatsby-theme-core@eshlox/gatsby-theme-axii@rodriguesmyron/gatsby-theme-podcast@hummingbird/gatsby-theme@riencoertjens/gatsby-theme-webhart-base@hesburgh-wse/gatsby-theme-marble@g.vocale/gatsby-theme-memoriestoriche@fox-tech/docpersonal-site@gatsbystorefront/gatsby-theme-storefront-shopify@giraldomac/gatsby-theme-mindmerge@giraldomac/gatsby-theme-mmdbase@giraldomac/gatsby-theme-wvumedicine@julien76/gatsby-theme-jhe@smplslaps/gatsby-theme-storefront-shopify@strv/gatsby-theme-core@wingscms/hummingbird@dungle-scrubs/gatsby-shopify-theme@cloudexible/gatsby-theme-storefront-shopify@codinators/gatsby-theme-blog
1.6.2

1 month ago

1.6.1

1 month ago

1.6.0

1 month ago

1.5.6

2 months ago

1.5.5

6 months ago

1.5.4

6 months ago

1.5.3

9 months ago

1.5.2

9 months ago

1.5.1

1 year ago

1.5.0

2 years ago

1.4.0

2 years ago

1.3.0

3 years ago

1.2.0

3 years ago

1.1.0

3 years ago

1.0.2

3 years ago

1.0.1

3 years ago

1.0.0

3 years ago