Exotic README

English | 简体中文 | 中国镜像

Exotic (representing Exotic Star) is a professional version of PulsarRPA, which contains a upgraded PulsarRPA server, a set of top e-commerce site scraping examples, and a applet for auto extraction supported by advanced AI.

Never write another web scraper. Exotic learns from the website, automatically generates all the extract rules, queries the Web as a database, and delivers web data completely and accurately at scale:

1. STEP1: automatically extract every field in webpages using advanced AI and generate extract SQLs

2. STEP2: test the SQLs and improve them to match frontend business requirements if necessary

3. STEP3: create crawl rules in the web console to run extract SQLs continuously and download all the web data to drive your business forward

There are already dozens of scraping cases for the most popular websites, we are constantly adding more cases.

• Web spider: browser rendering, ajax data crawling

• High performance: highly optimized, rendering hundreds of pages in parallel on a single machine without be blocked

• Low cost: scraping 100,000 browser rendered e-comm webpages, or n * 10,000,000 data point each day, only 8 core CPU/32G memory are required

• Web UI: a very simple yet powerful web UI to manage spiders and download data

• Machine learning: automatically extract every field in webpages using unsupervised machine learning and generate extract rules and SQLs

• Data quantity assurance: smart retry, accurate scheduling, web data lifecycle management

• Large scale: fully distributed, designed for large scale crawling

• Simple API: single line of code to scrape, or single SQL to turn a website into a table

• X-SQL: extended SQL to manage web data: Web crawling, scraping, Web content mining, Web BI

• Bot stealth: IP rotation, web driver stealth, never get banned

• RPA: simulating human behaviors, SPA crawling, or do something else awesome

• Big data: various backend storage support: MongoDB/HBase/Gora

• Logs & metrics: monitored closely and every event is recorded

• Memory 4G+

• The latest version of the Java 11 JDK

• Java and jar on the PATH

• Google Chrome 90+

Download the latest executable jar:

Add the following lines to your .m2/settings.xml.

For Chinese developers, we strongly suggest that you follow this instruction to accelerate the building.

Note: if you are using CMD or PowerShell on Windows, you may need to remove the wildcard * and use the full name of the jar.

If Exotic is running in GUI mode, the web console should open within a few seconds, or you can open it manually:

http://localhost:2718/exotic/crawl/

We can use the harvest command to leans from a set of item pages using unsupervised machine learning.

The URL in the command above should be an portal URL, such as the URL of the product listing page.

Exotic visits the portal URL, finds out the best link set for item pages, fetches item pages and then learn from them.

Here is a snapshot of the result of auto extract using unsupervised machine learning for an e-comm site.

The best CSS selectors for each field are generated automatically, you can use these rules for web scraping in the old-fashioned way:

And also the generated SQL:

Note that the website in this demo uses CSS obfuscation techniques, so the CSS selectors are hard to read and changes frequently. There is no other effective technology to solve this problem other than machine learning based solutions.

The complete code can be found here.

The harvest command extracts fields automatically using unsupervised machine learning, and also generates the best css selectors for all possible fields and the extract SQLs. We can execute the SQLs using sql command.

Run the executable jar directly for help to explore more power provided:

This command will print the help message and most useful examples.

Q: How to use proxies?

A: Follow this guide for proxy rotation.