@crawlora/sdk NPM

Crawlora SDK

The Crawlora SDK provides a simple and efficient way to interact with the Crawlora web crawling platform. It allows developers to manage crawlers, fetch data, and integrate crawling services into their applications using TypeScript.

Features

Easy integration with Crawlora's API.
Manage and schedule crawlers programmatically.
Fetch and process web data efficiently.
Promise-based API for asynchronous operations.

Installation

To install the SDK, use the following command with npm:

npm install crawlora

Quick Start

Initializing the SDK

To use the SDK, import it into your project and initialize it with your API Key:

import { Crawlora } from 'crawlora';

const client = new Crawlora({
    apiKey: 'YOUR_API_KEY',
});

Creating and Running a Crawler

client.createCrawler({
  name: 'example-crawler',
  startUrl: 'https://example.com',
  callbackUrl: 'https://yourapp.com/callback',
}).then(crawler => {
  console.log(`Crawler created: ${crawler.id}`);
}).catch(error => {
  console.error('Error creating crawler:', error);
});

Fetching Data

Once a crawler has completed its run, you can fetch the data:

client.getCrawlData('crawler-id')
  .then(data => {
    console.log(data);
  })
  .catch(error => {
    console.error('Error fetching crawl data:', error);
  });

API Reference

`createCrawler(options): Promise<Crawler>`

options: Object containing crawler settings.
- name: Crawler name.
- startUrl: The URL to start crawling.
- callbackUrl: URL to receive the crawled data.
Returns: A promise that resolves to the created Crawler object.

`getCrawlData(crawlerId: string): Promise<any>`

crawlerId: The ID of the crawler whose data you want to retrieve.
Returns: A promise that resolves to the crawled data.

Configuration

You can configure the SDK using environment variables or directly within your application.

Environment Variables

CRAWLORA_API_KEY: Your API Key for authentication.

Direct Configuration

Pass options during initialization:

const client = new Crawlora({
    apiKey: 'YOUR_API_KEY',
    timeout: 5000, // Timeout for API requests
});

Error Handling

The SDK provides built-in error handling for network issues, API rate limits, and other common errors. Example:

client.getCrawlData('crawler-id')
  .then(data => {
    console.log(data);
  })
  .catch(error => {
    console.error('Error fetching crawl data:', error);
  });