@carbondigital/robots-parser NPM

About The Project

We wanted a tool internally that would help us rapidly scale our SEO efforts, but ultimately, give us the ability to migrate offf our current toolchain for SEO. Like many agencies, our team spent thousands on SEO-realted software each year. Since we're not a multi-million dollar agency, it always hurts to pay those bills.

In light of that, we are building this software initially for our own team. We have already seen the power of competing with well-known tools that currently exist. One thing we aim to achieve is more honesty and more transparency. This is why we are building some modules has open-source NPM packages. So, give all developers the ability to see exactly what we're doing and contribute if they desire.

Built With

Below are the major frameworks/libraries that are currently used in this project, or soon to be added in a future release.

Usage

Console.Log your robots.txt file:

import { robotsParser } from '@carbondigital/robots-parser';
robotsParser('console', './my-robots-file.txt');

Logged Output:

Sitemap: https://carbondigital.us/sitemaps/post-sitemap1.xml
Sitemap: https://carbondigital.us/sitemaps/page-sitemap1.xml
Sitemap: https://carbondigital.us/sitemaps/projects-sitemap1.xml
Sitemap: https://carbondigital.us/sitemaps/services-sitemap1.xml
Sitemap: https://carbondigital.us/sitemaps/locations-sitemap1.xml
Sitemap: https://carbondigital.us/sitemaps/category-sitemap1.xml

User-agent: SemrushBot
    Disallow: /
User-agent: SemrushBot-SA
    Disallow: /
User-agent: MJ12bot
    Disallow: /
User-agent: Mediapartners-Google
    Disallow:
User-agent: Googlebot-Image
    Disallow:
User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

Return Array of data from your robots.txt file:

import { robotsParser } from '@carbondigital/robots-parser';
robotsParser('browser', './my-robots-file.txt');

Array Output:

[
  {
    type: 'sitemap',
    content: 'https://carbondigital.us/sitemaps/post-sitemap1.xml'
  },
  {
    type: 'sitemap',
    content: 'https://carbondigital.us/sitemaps/page-sitemap1.xml'
  },
  {
    type: 'sitemap',
    content: 'https://carbondigital.us/sitemaps/projects-sitemap1.xml'
  },
  {
    type: 'sitemap',
    content: 'https://carbondigital.us/sitemaps/services-sitemap1.xml'
  },
  {
    type: 'sitemap',
    content: 'https://carbondigital.us/sitemaps/locations-sitemap1.xml'
  },
  {
    type: 'sitemap',
    content: 'https://carbondigital.us/sitemaps/category-sitemap1.xml'
  },
  { type: 'blank', content: '' },
  { type: 'user-agent', content: 'SemrushBot' },
  { type: 'disallow', content: ' /' },
  { type: 'user-agent', content: 'SemrushBot-SA' },
  { type: 'disallow', content: ' /' },
  { type: 'user-agent', content: 'MJ12bot' },
  { type: 'disallow', content: ' /' },
  { type: 'user-agent', content: 'Mediapartners-Google' },
  { type: 'disallow', content: '' },
  { type: 'user-agent', content: 'Googlebot-Image' },
  { type: 'disallow', content: '' },
  { type: 'user-agent', content: '*' },
  { type: 'disallow', content: ' /wp-admin/' },
  { type: 'allow', content: ' /wp-admin/admin-ajax.php' }
];

Our SEM Links app is a Nest.js project. In that project, we have a data wrapper so all returned data is part of a "data" object. When testing this in Postman, here is the same data being returned.

{
  "data": [
    {
      "type": "sitemap",
      "content": "https://carbondigital.us/sitemaps/post-sitemap1.xml"
    },
    {
      "type": "sitemap",
      "content": "https://carbondigital.us/sitemaps/page-sitemap1.xml"
    },
    {
      "type": "sitemap",
      "content": "https://carbondigital.us/sitemaps/projects-sitemap1.xml"
    },
    {
      "type": "sitemap",
      "content": "https://carbondigital.us/sitemaps/services-sitemap1.xml"
    },
    {
      "type": "sitemap",
      "content": "https://carbondigital.us/sitemaps/locations-sitemap1.xml"
    },
    {
      "type": "sitemap",
      "content": "https://carbondigital.us/sitemaps/category-sitemap1.xml"
    },
    {
      "type": "blank",
      "content": ""
    },
    {
      "type": "user-agent",
      "content": "SemrushBot"
    },
    {
      "type": "disallow",
      "content": "/"
    },
    {
      "type": "user-agent",
      "content": "SemrushBot-SA"
    },
    {
      "type": "disallow",
      "content": "/"
    },
    {
      "type": "user-agent",
      "content": "MJ12bot"
    },
    {
      "type": "disallow",
      "content": "/"
    },
    {
      "type": "user-agent",
      "content": "Mediapartners-Google"
    },
    {
      "type": "disallow",
      "content": ""
    },
    {
      "type": "user-agent",
      "content": "Googlebot-Image"
    },
    {
      "type": "disallow",
      "content": ""
    },
    {
      "type": "user-agent",
      "content": "*"
    },
    {
      "type": "disallow",
      "content": "/wp-admin/"
    },
    {
      "type": "allow",
      "content": "/wp-admin/admin-ajax.php"
    }
  ]
}

Roadmap

We've got a simple setup on Canny. You can submit either a Github issue, or a Canny feature request for the SEM Links tool.

Feature Requests in Canny

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again! See contributing.md for more details.