puppeteer-extra-plugin-jquery v0.3.12
puppeteer-extra-plugin-jquery
A plugin for puppeteer-extra to use jQuery selector in puppeteer
Install
yarn add puppeteer-extra-plugin-jquery
# - or -
npm install puppeteer-extra-plugin-jquery
If this is your first puppeteer-extra plugin here's everything you need:
yarn add puppeteer puppeteer-extra puppeteer-extra-plugin-jquery
# - or -
npm install puppeteer puppeteer-extra puppeteer-extra-plugin-jquery
Usage Javascript
// puppeteer-extra is a drop-in replacement for puppeteer,
// it augments the installed puppeteer with plugin functionality
const puppeteer = require('puppeteer-extra')
// add jquery plugin
const jQueryPlugin = require('puppeteer-extra-plugin-jquery/lib/injector');
puppeteer.use(jQueryPlugin())
// puppeteer usage as normal
puppeteer.launch({ headless: true }).then(async browser => {
console.log('Running tests..')
const page = await browser.newPage()
await pageX.goto('https://github.com/UrielCh/puppeteer-jquery/tree/master/puppeteer-jquery', { waitUntil: 'domcontentloaded' })
// use waitForjQuery()
const start = await pageX.waitForjQuery('span.Counter.js-social-count');
console.log('selector match ', start.length, 'elements');
// use any jQuery code.
const cnt = await pageX.jQuery('span.Counter.js-social-count').text();
console.log('this project have', cnt, 'start');
await browser.close();
}
Usage TypeScript
puppeteer-extra-plugin-jquery
is intend to be use in TS to unleash its full potential Typing is available for most jQuery syntax.
import puppeteer from 'puppeteer-extra'
import { Plugin, PageEx } from 'puppeteer-extra-plugin-jquery'
puppeteer
.use(new Plugin())
.launch({ headless: true })
.then(async browser => {
const pageOrg = await browser.newPage()
// cast page as PageEx (will pe update with future puppeteer-extra version)
const page = pageOrg as unknown as PageEx;
await page.goto('https://github.com/UrielCh/puppeteer-jquery/tree/master/puppeteer-jquery', { waitUntil: 'domcontentloaded' })
const start = await page.waitForjQuery('span.Counter.js-social-count');
console.log('selector match ', start.length, 'elements');
const cnt = await page.jQuery('span.Counter.js-social-count').text();
console.log('this project have', cnt, 'start');
await browser.close();
})
This plugin fullname is puppeteer-extra-plugin-jquery/lib/Plugin
Old doc
Usage typescript
Handle Page
instance as PageEx
, and get access to page.jQuery(selector: string)
npm install puppeteer
npm install puppeteer-jquery
npm install --save-dev typescript @types/node ts-node
import puppeteer from 'puppeteer';
import { pageExtend } from 'puppeteer-jquery';
(async () => {
let browser = await puppeteer.launch({headless: true});
let pageOrg = await browser.newPage();
let page = pageExtend(pageOrg);
await page.jQuery('body').append(`<h1>Title</h1> <div><h3>sub-title <i>X</i><h3> <h4>h4</h4></div>`);
// get the H1 value
let title = await page.jQuery('h1').text();
// chain calls
let text = await page.jQuery('body i:last')
.closest('div')
.find('h3')
.css('color', 'yellow')
.parent()
.find(':last')
.text();
console.log('this page contains H1:', title);
console.log('last h4 contains', text);
})();
Usage javascript
Handle Page
instance as PageEx
, and get access to page.jQuery(selector: string)
npm install puppeteer
npm install puppeteer-jquery
const puppeteer = require ('puppeteer');
const { pageExtend } = require('puppeteer-jquery');
(async () => {
let browser = await puppeteer.launch({headless: true});
let pageOrg = await browser.newPage();
let page = pageExtend(pageOrg);
await page.jQuery('body').append(`<h1>Title</h1> <div><h3>sub-title <i>X</i><h3> <h4>h4</h4></div>`);
// get the H1 value
let title = await page.jQuery('h1').text();
// chain calls
let text = await page.jQuery('body i:last')
.closest('div')
.find('h3')
.css('color', 'yellow')
.parent()
.find(':last')
.text();
console.log('this page contains H1:', title);
console.log('last h4 contains', text);
})();
Advanced common usage typescript
npm install puppeteer
npm install puppeteer-jquery
npm install --save-dev typescript @types/node ts-node
import puppeteer from 'puppeteer';
import { pageExtend } from 'puppeteer-jquery'
(async() =>{
let browser = await puppeteer.launch({headless: true});
let pageOrg = await browser.newPage();
await page.goto('http://maywebsite.abc', {
waitUntil: 'networkidle2',
});
let page = pageExtend(pageOrg);
// get all li text in the page as an array
const data: string[] = await jqPage
.jQuery('li')
.map((id: number, elm: HTMLElement) => elm.textContent)
.pojo();
})();
data
contains somethink like:
[ "a mug", "a hat"]
Advanced common usage javascript
npm install -g typescript @types/node ts-node
npm init -y
npm install puppeteer puppeteer-jquery picocolors
npm --save-dev install @types/jquery
Fill tsconfig.json:
{
"compilerOptions": {
"target": "es2017",
"lib": [ "DOM", "ES2017" ],
"types": [ "node", "jquery" ],
"module": "commonjs",
"esModuleInterop": true,
"strict": true,
}
}
Fill your the code in index.ts
import puppeteer from 'puppeteer';
import { pageExtend } from 'puppeteer-jquery';
import pc from 'picocolors';
import type jq from 'jquery'
var jQuery: typeof jq;
(async() =>{
let browser = await puppeteer.launch({headless: false});
let pageOrg = await browser.newPage();
await pageOrg.goto('https://github.com/UrielCh/puppeteer-jquery', { waitUntil: 'networkidle2' });
let jqPage = pageExtend(pageOrg);
const stars: string = await jqPage.jQuery('#repo-stars-counter-star').text();
console.log(`my project is only ${pc.yellow(stars)}⭐`);
const files = await jqPage.jQuery('div[aria-labelledby="files"] > div[role="row"].Box-row')
.map((id: number, elm: HTMLElement) => {
const div = jQuery(elm);
const icon = (div.find('[role="gridcell"] [aria-label]:first').attr('aria-label') || '').trim();
const filename = (div.find('div[role="rowheader"]').text() || '').trim();
const lastChange = (div.find('[role="gridcell"]:last').text() || '').trim();
return {icon, filename, lastChange};
}).pojo<{icon: string, filename: string, lastChange: string}>();
for (const file of files) {
console.log(`file ${pc.green(file.filename)} is ${file.icon} had been change ${file.lastChange} `);
}
browser.close()
})();
ts-node index.ts
my project is only 3219⭐
file .vscode is Directory had been change 13 months ago
file playwright-jquery is Directory had been change 4 months ago
file puppeteer-jquery is Directory had been change 4 months ago
file .gitignore is File had been change 3 years ago
file LICENSE is File had been change 3 years ago
file README.md is File had been change 13 months ago
Advanced common usage javascript
npm init -y
npm install puppeteer puppeteer-jquery picocolors
npm --save-dev install @types/jquery
Fill your the code in index.js
const puppeteer = require('puppeteer');
const { pageExtend } = require('puppeteer-jquery');
const pc = require('picocolors');
var jQuery;
(async() =>{
let browser = await puppeteer.launch({headless: false});
let pageOrg = await browser.newPage();
await pageOrg.goto('https://github.com/UrielCh/puppeteer-jquery', { waitUntil: 'networkidle2' });
let jqPage = pageExtend(pageOrg);
/** @type {string} */
const stars = await jqPage.jQuery('#repo-stars-counter-star').text();
console.log(`my project is only ${pc.yellow(stars)}⭐`);
const files = await jqPage.jQuery('div[aria-labelledby="files"] > div[role="row"].Box-row')
.map((id, elm) => {
const div = jQuery(elm);
const icon = (div.find('[role="gridcell"] [aria-label]:first').attr('aria-label') || '').trim();
const filename = (div.find('div[role="rowheader"]').text() || '').trim();
const lastChange = (div.find('[role="gridcell"]:last').text() || '').trim();
return {icon, filename, lastChange};
}).pojo();
for (const file of files) {
console.log(`file ${pc.green(file.filename)} is ${file.icon} had been change ${file.lastChange} `);
}
browser.close()
})();
ts-node index.ts
output:
my project is only 3220⭐
file .vscode is Directory had been change 13 months ago
file playwright-jquery is Directory had been change 4 months ago
file puppeteer-jquery is Directory had been change 4 months ago
file .gitignore is File had been change 3 years ago
file LICENSE is File had been change 3 years ago
file README.md is File had been change 13 months ago
Usage Mixed with puppeteer-extra
npm init -y
npm install puppeteer-extra puppeteer-extra-plugin-stealth puppeteer
import { pageExtend, PageEx } from 'puppeteer-jquery'
import puppeteer from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth'
puppeteer.use(StealthPlugin())
const page1 = 'https://recaptcha-demo.appspot.com/recaptcha-v3-request-scores.php';
const main = async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
const pageEx: PageEx = pageExtend(page);
await page.goto(page1, { waitUntil: 'domcontentloaded' }); // 'networkidle0'
const go = await pageEx.waitForjQuery('button.go');
if (!go.length) {
console.error('go button not found');
return;
}
await pageEx.jQuery('button.go').map((index, element) => { jQuery(element).trigger('click'); })
await page.$eval('button.go', (el: Element) => (el as HTMLElement).click());
const r1 = await pageEx.waitForjQuery('pre.response:contains("score")');
console.log(await r1[0].boundingBox());
await page.screenshot({ path: 'testresult.png', fullPage: true })
const result = await pageEx.jQuery('pre.response').text();
console.log('score is:' + JSON.parse(result).score);
await page.waitForTimeout(500);
await page.close();
await browser.close();
}
main();
Notes
You may also install @types/jquery
dependence for more complex JQuery task, in this case always use jQuery
method, do not use $
sortcut, the bundeled jQuery will be renamed before being injected. the injection process rename fullname jQuery
to the rigth value before injections.
changelog
- V0.3.12 add each() fix puppeteer-extra compatibility
- V0.3.11 fix some first run error
- V0.3.10 exec() and pojo() accept arguments
- V0.3.9 fix jQuery.filter() signature
- V0.3.8 update typing
- V0.3.7 add getJQueryName(), jQuery.filter(), add onTimeout options
- V0.3.6 update docs
- V0.3.5 project renamed to puppeteer-extra-plugin-jquery and can be use with berstend/puppeteer-extra
- V0.3.3 improve typing + update all deps
- V0.3.0 Add a advance example in doc, improve map signature, add not(), offsetParent(), update is(), add scrapping test, unify code to work with playwright
- V0.2.1 Add a advance example in doc, improve map signature, add not(), offsetParent(), update is(), add scrapping test.
- V0.2.0 project backmto live, puppeter is now writen in typescript, add some jquery method (attr(string), css(string), prop(string))
- V0.1.8 change waitForjQuery return type to ElementHandle[]
- V0.1.7 add waitForjQuery
- V0.1.6 update doc
around this project
- melbourne2991/jquery-puppeteer Simple JQuery integration, by adding a
page.evalJquery()
. - playwright-jquery The playwright version.