Skip to content

Latest commit

 

History

History
45 lines (34 loc) · 2.03 KB

README.md

File metadata and controls

45 lines (34 loc) · 2.03 KB

Panther

A distributed web crawler written in python.

See my post for more info.

Forks Stargazers Issues MIT License LinkedIn

Requirements

Docker for Desktop

Usage

git clone this repo

Run docker-compose build on your machine. Once the process is finished, run docker-compose up to start the service.

URLs can be added to the frontier-queue via localhost:5000, and workers can be monitored via localhost:5001. Finally, the corpus of documents can be monitored via localhost:5002.

Contributing

  1. Fork it (https://github.com/yourname/yourproject/fork)
  2. Create your feature branch (git checkout -b feature/fooBar)
  3. Commit your changes (git commit -am 'Add some fooBar')
  4. Push to the branch (git push origin feature/fooBar)
  5. Create a new Pull Request