Stanley Crawler With Events By Dragons Gamers

Template Stanley for Crawler Puppeteer/Playwright project !

Table of Contents

• Benefits
• Libraries
• Dependencies
• Use Template
  • Change to Puppeteer
• Get Started
  • Event / Listeners
  • Selector
  • Container Bind
  • Extract Text Helper
  • Make File
• Cycle Time
  • Project Life Cycle
  • Service Life Cycle
  • Page Life Cycle
  • Handler Life Cycle
• Prepare to develop
• Start Project
• Build and Run
• Teste Code

---

Benefits

• Speed start new crawler using typescript
• Over 800 rules for pattern, possible errors and errors in Linter
• Code quality guaranteed
• AutoReview when opening a pull-request/merge
• Automatic Test when opening pull-request/merge
• Automatic Package and release generate on merge
• IOT/IOC (Inversion of Control) for easy use of libraries
• Make:Files Command to make base files

Libraries

• Node.js 18
• Typescript
• Eslint
• ODG-Linter-JS
• EditorConfig
• ReviewDog

Dependencies

• Node.js 18 or later
• Yarn Optional/Recommended
• Playwright
  • For web scraping / crawling / automation
• @odg/axios
  • Example to message request with axios
• @odg/chemical-x
  • Base For crawlers and helpers project
• @odg/command
  • Commands to make:files
• @odg/events
  • Abstract Events with async events
• @odg/exception
  • Create custom exception to debug and history
• @odg/json-log
  • Convert your logs exception, request, in JSON before send
• @odg/log
  • Interface of logs
• @odg/message
  • Interface of request and messages
• @odg/tsconfig
  • Base Tsconfig to build project

Use Template

Click in use this template button and clone your template project

Change to Puppeteer

To change you crawler to use puppeteer you change ./engine.ts file for:

import puppeteer, {
    type Browser,
    type PuppeteerLaunchOptions,
    type Page,
    type BrowserContext,
    type PuppeteerNode,
} from "puppeteer";

export type BrowserTypeEngine = PuppeteerNode;

export type BrowserClassEngine = Browser;

export type ContextClassEngine = BrowserContext;

export type PageClassEngine = Page;

export type BrowserOptionsEngine = PuppeteerLaunchOptions;

export const browserEngine = puppeteer;

Get Started

---

Event / Listeners

Events can be fired with EventBusInterface Example Include the execution of your page in an event so that you can execute any page at any time

export class ExampleEventListener implements EventListenerInterface<EventTypes, EventName.ExampleEvent> {

    @inject(ContainerName.Logger)
    public readonly log!: LoggerInterface;

    public async handler(_receivedData: unknown) {
        await this.log.info('Example Event Listener');
    }

}

Register all Events name in ./src/app/Enums/EventName.ts

---

Register all Events Types in ./@types/EventsInterface.ts

---

Register your events in EventServiceProvider to automatic listeners.

Selector

The selectors are in the selectors folder

export const googleSearchSelector = {
    searchInput: "input[name=\"q\"]",
    buttons: {
        submit: "input[name=\"btnK\"] >> visible=true",
    },
};

export type GoogleSearchSelectorType = typeof googleSearchSelector;

Register all Selectors in ./src/app/Selectors/index.ts

Container Bind

this.bind(
    ContainerName.Logger,
).to(ConsoleLogger).inSingletonScope();

Register all Container name in src/app/Enums/ContainerName.ts Register all Container Inject in src/app/Container.ts

Extract Text Helper

Use Str, Num, Arr helpers class to manipulate Text, Numbers and Arrays

import { Str } from "@odg/chemical-x";

const value = new Str(
    this.page.locator("#money").innerText(),
).money();

Make File

To Make a file run command

Use this to make Page, Handler, Selectors...

yarn odg make

Example to make Page File

yarn odg make:page Search

# help with
yarn odg make:page --help

Cycle Time

Project Life Cycle

Project Life cycle

Service Life Cycle

Example Service Life Cycle

Page Life Cycle

Example Page Life Cycle

Handler Life Cycle

Example Handler Life Cycle

Prepare To Develop

Copy .env.example to .env and add the values according to your needs.

Start Project

First install dependencies with the following command

yarn install
# or
npm install

Build and Run

To build the project, you can use the following command

if you change files, you need to run yarn build and yarn start again

yarn build && yarn start
# or
yarn dev

Teste Code

To Test execute this command

yarn test
# or
yarn test:watch