Universal scraping tool, which allows you to extract data using multiple environments
This tool moves routine crawling process to the new level. Now it's possible to parse a web page for a moment. All you need is to specify parsing rules based on css selectors. It's so simple as Goose can do it. This library allows to parse such data types as Grid, Collections, and Simple objects. Parser has support of pagination by extension goose-paginator. Also it offers you following features: actions to interact with the page and transforms to convert parsed data to friendly format.
Now it's easy to start with Goose, just try to use goose-starter-kit for it.
yarn add goose-parser goose-chrome-environment
const Parser = require('goose-parser');
const ChromeEnvironment = require('goose-chrome-environment');
const env = new ChromeEnvironment({
url: 'https://www.google.com/search?q=goose-parser',
});
const parser = new Parser({ environment: env });
(async function () {
try {
const results = await parser.parse({
actions: [
{
type: 'wait',
timeout: 10 * 1000,
scope: '.srg>.g',
parentScope: 'body'
}
],
rules: {
scope: '.srg>.g',
collection: [[
{
name: 'url',
scope: 'h3.r>a',
attr: 'href',
},
{
name: 'text',
scope: 'h3.r>a',
}
]]
}
});
console.log(results);
} catch (e) {
console.log('Error occurred:');
console.log(e.stack);
}
})();
This is a special atmosphere where Parser has to be executed. The main purpose of an environment is to provide a method for evaluating JS on the page. Goose supports following environments:
For now it's available to run goose-parser as a docker service.
Params:
Options:
There are two options to run it:
docker run -it --rm -e "DEBUG=*,-puppeteer:*" redcode/goose-parser:chrome-1.1.3-parser-0.6.0\
https://www.google.com/search?q=goose-parser\
'{
"actions": [
{
"type": "wait",
"scope": ".g"
}
],
"rules": {
"scope": ".g",
"collection": [
[
{
"scope": ".r>a h3",
"name": "name"
},
{
"scope": ".r>a:eq(0)",
"name": "link",
"attr": "href"
}
]
]
}
}'
Create a file rules/rules.json
which contains parser rules and run following command:
docker run -it --rm --volume="`pwd`/rules:/app/rules:ro" -e "DEBUG=*,-puppeteer:*" redcode/goose-parser:chrome-1.1.3-parser-0.6.0 --rules-file="/app/rules/rules.json" 'https://www.google.com/search?q=goose-parser'
Based on the code you can find detailed documentation about actions and transformations
API reference - coming soon