# tecrawl

> 基于Crawler和temme的静态网页抓取工具

Latest version **1.0.2** (published 2023-08-06) · ISC license · 0 weekly downloads

> **Deprecated.** This package is deprecated.

## Install

```sh
npm install tecrawl
pnpm add tecrawl
yarn add tecrawl
bun add tecrawl
```

## Health

**Score 10/100 (F)** — status: deprecated.

Negative: deprecated.

## Facts

| | |
|---|---|
| Version | 1.0.2 |
| Published | 2023-08-06 |
| First published | 2022-08-30 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 4 |
| Unpacked size | 9.7 KB |
| Known vulnerabilities | 0 (+23 in 1 direct dependencies) |
| Install scripts | no |
| GitHub stars | 0 |
| Author | halfong |
| Maintainers | sunkey |
| Keywords | 抓取, nodejs |

## Links

- npm: https://www.npmjs.com/package/tecrawl
- Repository: https://github.com/sunkeyfong/tecrawl
- Homepage: https://github.com/sunkeyfong/tecrawl#readme
- Issues: https://github.com/sunkeyfong/tecrawl/issues
- npm.io page: https://npm.io/package/tecrawl

## Dependencies (4)

- [axios](https://npm.io/package/axios.md) ^0.21.1
- [temme](https://npm.io/package/temme.md) ^0.8.3
- [moment](https://npm.io/package/moment.md) ^2.29.4
- [crawler](https://npm.io/package/crawler.md) ^1.3.0

## Recent versions

- 1.0.2 (latest) — 2023-08-06
- 1.0.1 — 2022-08-30
- 1.0.0 — 2022-08-30

## README

## 使用限制

tecrawl是一个轻量级的抓取脚本，通常只适合抓取**无防抓策略**的**静态**网页内容。

## 使用方法

```javascript
const { Tecrawl } = require('tecrawl');
const $tec = new Tecrawl([
    {
      name : 'XXX网站',
      /**
       * 参数 queue
       * 相同HTML模板的多个url，依次抓取
       */
      queue : [
          'http://www.ctnews.com.cn/news/node_1.html',
          'http://www.ctnews.com.cn/news/node_2.html',
      ],
      /**
       * 参数 tm
       * 解析此模板的temme语法，详见temme库文档
       */
      tm :`
          li.list__item@{
              .list__img img[src=$image];
              .list__title a[href=$link]{ $title }
              .list__summary a{ $teaser }
              $source='中国旅游新闻网';
          };
      `,
      /**
       * 参数 format
       * 抓取后数据处理
       */
      format : items => items.map( i => {
          i.date = i.link.replace(/.*(\d\d\d\d-\d\d)\/(\d\d).*/,"$1-$2")
          return i
      }),
    }
    // ... 可以添加产出同类数据的多个source模板
  ]);

  // 执行抓取 => $tec.threads
  await $tec.crawl();
  console.log( '抓取结果：' , $tec.threads );

  // 便捷函数： 根据thread字段排重
  $tec.unique('title');

  // 边界函数：根据thread.date区间过滤
  $tec.dateBetween( sdate , edate );
```



## Credits

![Temme - 很有创意的js-html解析库](https://github.com/shinima/temme)
![Crawler - 耐用的nodejs-http抓取库](https://github.com/bda-research/node-crawler)

---
_Source: https://npm.io/package/tecrawl · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._