1. install: pip3 install scrapy

2. Run this command in your terminal to download the custom scrapy spider, ReconSpider, and extract it to the current working directory.

        shellsession
Moyinthehacker@htb[/htb]$ wget -O ReconSpider.zip https://academy.hackthebox.com/storage/modules/144/ReconSpider.v1.2.zip
Moyinthehacker@htb[/htb]$ unzip ReconSpider.zip

1. With the files extracted, you can run ReconSpider.py using the following command:

        shellsession
Moyinthehacker@htb[/htb]$ python3 ReconSpider.py http://inlanefreight.com/

N.B. Dont forget to add the /, The trailing slash tells the program that the URL is a directory/root, not a file.

1. Results are saved to results.json

use in kalli : source ~/venvs/scrapy/bin/activate

end in kali: deactivate