@studyportals/fawkes v7.0.1

fawkes

This project is used to centralize indexability logic for our Search microservices and Sitemap Generator. The main purpose is to completely remove discrepancies between pages indexed by Search and page URLs stored in Sitemap.

Link to documentation: https://studyportals.atlassian.net/wiki/x/BwAdvw

Base structure

• organisations-seo (used by Organisation Search)
• scholarships-seo (used by Scholarship Search)
• sitemap-generator-seo (used by Sitemap Generator)
• src
  • common
  • organisations
  • scholarships
  • sitemap-generator

Automatic Documentation System

Overview

This system automatically generates and maintains documentation for Fawkes indexing rules across multiple projects. It creates a hierarchical structure of Confluence pages that document all indexing policies and rules, ensuring that the documentation is always up-to-date with the actual codebase.

Link to documentation: Fawkes Indexing Rules Documentation

Features

• Dynamic Documentation: Automatically extracts indexing rules and policies directly from the codebase
• Multi-Page Structure: Creates a main index page with separate pages for each project
• Hierarchical Organization: Structures content as Projects → Policies → Rules
• Automatic Page Creation: Intelligently creates or updates pages as needed
• Cross-Page Navigation: Provides intuitive navigation between all pages

Architecture

The system follows SOLID design principles to ensure maintainability and extensibility:

• Single Responsibility: Each class has one specific responsibility
• Open/Closed: Easily extendable without modifying existing code
• Liskov Substitution: Interchangeable components through well-defined interfaces
• Interface Segregation: Clean, focused interfaces
• Dependency Inversion: High-level modules depend on abstractions

Key Components

1. Content Builder: Extracts project, policy, and rule data from the Fawkes system
2. Policy Fetchers: Project-specific components that fetch and format policy data
3. Document Builder: Generates structured HTML content for Confluence pages
4. Confluence Updater: Handles creation and updating of Confluence pages
5. Token Retriever: Securely retrieves API token for Confluence access

How It Works

1. The system starts by collecting all project data using the ContentBuilder
2. For each project, it extracts policies and rules using project-specific fetchers
3. The DocumentBuilder generates the content for the main index page and each project page
4. The ConfluenceUpdater creates or updates pages in Confluence as needed
5. Each project page links back to the main index, and the main index links to all project pages

Setup and Configuration

Environment Variables

The system requires the following environment variables:

CONFLUENCE_EMAIL=your-email@example.com
CONFLUENCE_API_TOKEN=your-api-token
CONFLUENCE_BASE_URL=https://your-instance.atlassian.net
CONFLUENCE_SPACE_KEY=YOUR_SPACE_KEY
CONFLUENCE_MAIN_PAGE_TITLE=Fawkes Indexing Rules Documentation

Important Note on Credentials: Currently, the system uses personal credentials (email and API token) for Confluence access. This is a temporary solution and should be replaced with a team or service account in the future to avoid dependency on a specific individual's account.

Deployment and Triggering

The system runs as an AWS Lambda function and is automatically triggered as part of the CI/CD pipeline. This ensures documentation is always updated whenever new code is deployed:

# buildspec.yml (excerpt)
post_build:
  commands:
    - |
      if [ -z "$PR_ID" ]; then
        LAMBDA_NAME="FawkesConfluenceUpdater-$STAGE"
        echo "Invoking Lambda function $LAMBDA_NAME"
        echo '{"source":"cicd-pipeline","timestamp":"'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'"}' > payload.json
        aws lambda invoke --function-name $LAMBDA_NAME --invocation-type Event --payload fileb://payload.json response.json || echo "Lambda invocation failed, but continuing"
      fi

Trigger Flow:

1. When code is pushed to the main branch, the CI/CD pipeline is triggered
2. After successful deployment of a new Fawkes version, the post-build phase executes
3. The Lambda function is invoked with a simple payload containing the source and timestamp
4. The Lambda function authenticates with Confluence and executes the documentation generation process
5. This only happens for production deployments (not PR builds)

This automated approach ensures documentation stays in sync with the actual codebase without manual intervention.

Scalability and Extension

Adding New Projects

To add documentation for a new project:

1. Create a new class that implements the IPolicyFetcher interface
2. Implement the getProjectName() and fetchPolicies() methods
3. Register the new fetcher in the ContentBuilder constructor

Example:

export class NewProjectPolicyFetcher implements IPolicyFetcher {
  getProjectName(): string {
    return 'New Project Name';
  }

  fetchPolicies(): IPolicy[] {
    // Implement policy fetching logic
    return [...policies];
  }
}

// Then update ContentBuilder:
constructor(fetchers?: IPolicyFetcher[]) {
  this.policyFetchers = fetchers || [
    // Existing fetchers
    new NewProjectPolicyFetcher()
  ];
}

Customizing Page Content

The DocumentBuilder class contains methods for generating page content. To customize the format:

1. Modify the buildMainDocument() method for changes to the main index page
2. Modify the buildProjectDocument() method for changes to project pages
3. Modify the buildPolicySection() and buildRulesTable() methods for changes to policy and rule sections

Conclusion

This documentation system ensures that Fawkes indexing rules are always well-documented and accessible to the team. By automatically generating documentation directly from the codebase, it eliminates the risk of documentation becoming outdated or inconsistent with the actual implementation.