Завершено

Scrapy/Selenium (Python) to extract texts and files(specific webpages) based on keywords.

I need to make a Python script (Scrapy or Selenium, I am up to suggestions) to extract information within some specific(I have around 12) websites - daily(auto) or manually.

The pages are in portuguese, but I can guide you into the key input-fields and key-pages to look for.

1. User input:

- Time period (if the page has this feature)

- Website to scrap.

- Keywords(can be a list of words) to look for.

- User chooses the local path to the files to be downloaded.

2. Back-end:

- Access the page.

- Searches the tabs that can have useful information(I will provide the specific parts for each webpage to make the queries) links inside that domain.

- Download(if the page serves files in doc, html or pdf) and look for the keywords.

- Extract all the related content (files or the text in html).

- Go around Captchas(if the page has captcha)

3. Logging:

- All the extracted content must have the URL which the information/file is available in the webpage - can be done by logs.

- All the extracted content must have the DATE which the scrapping has been made - can be done by logs.

4. Configuration:

- All key-fields (like CSSSelector for a date field) should be configurable for each spider.

- The URL to start scrapping each webpage should be configurable.

- If page contains Authentication(Login/Password), user will fill the configuration for it.

IMPORTANT:

1. My plan is to pay for each 4 mapped websites (so total project is for 3 "packs" of websites)

2. The content in few cases will need to be extracted from images.

3. Start your bid with the word forward, so I can know if you did read all the description.

4. If you can't extract properly the content I can give you another one to replace that one, so you still need to deliver 4 websites per milestone.

5. I WILL RELEASE THE MILESTONES ONLY AFTER YOU SEND ME THE CODE AND I AM TOTALLY SATISFIED (I WILL RUN TESTS TO CHECK FUNCTIONALITY).

I have many projects at hand and would be great to stablish a good relation with you, since I constantly need someone to work with me.

Thank you.

Квалификация: Data Scraping, Python, Scrapy, Selenium Webdriver, Веб-скрейпинг

Показать больше scrapy examples, python scrapy example, scrapy vs selenium, python web scraping, scrapy python 3, scrapy documentation, scrapy vs beautifulsoup, web scraping, extract dbx files, extract 3gp files, mapguide enterprise extract shp files, extract xml files website, python script text files, extract embedded files doc, extract bkf files systools bkf repair tool, testsuite example selenium python, extract perl files server, extract ole files rich text, extract mht files, test suites selenium python

О работодателе:
( 5 отзыв(-а, -ов) ) FORTALEZA, Brazil

ID проекта: #19212037

Поручен:

etuannv

Hi there, I am interested in your project. I would approach your project by using Python with Scrapy. The website will be written in Python with Django. Here is a demo project: Price tracking system: https://etuannv.c Больше

$250 USD за 10 дней(-я)
(63 отзывов(-а))
6.2

36 фрилансеров(-а) в среднем готовы выполнить эту работу за $579

Vlzinch

Hi! I’m experienced Python developer, and web-scraping is one of my main fields of knowledge, so I’m 100% confident that I can complete your project and extract data from the sites you need. Please contact me to d Больше

$748 USD за 7 дней(-я)
(61 отзывов(-а))
7.7
mhmhz

Hi Can you provide the sites so i can analysis them? Thanks

$800 USD за 5 дней(-я)
(103 отзывов(-а))
7.4
zhangyingtai

forward Hello sir I have 9 years of experience about web scraping and have made 200+ crawlers with python. I have fully understood the project and I am confident. I can start the work right now. Best Regards, Больше

$588 USD за 10 дней(-я)
(111 отзывов(-а))
7.5
$1000 USD за 7 дней(-я)
(93 отзывов(-а))
7.2
zekovicm

Forward Hi there,I am Python Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this project ! I can start immediately and fi Больше

$705 USD за 10 дней(-я)
(91 отзывов(-а))
7.2
polarjin2017

Here is my selenium with python working result. [login to view URL] python selenium web driver app to scrap live data from the web site and export to excel file. This is just what I've done. I can do pytho Больше

$250 USD за 3 дней(-я)
(49 отзывов(-а))
6.4
dreammate0621

Hello! Let's just rest a moment. <Actions speak louder than words!> Nice to meet You! I am a WEB expert! I am interested in Your project. I wanna work with You. If you hire me, I am gonna do my best for Your proj Больше

$555 USD за 10 дней(-я)
(5 отзывов(-а))
6.3
C3guru

forward I've read your requirements about User Input,Back-end,Logging and Configuration. I have a good experience with selenium and python. Recently,I've developed B*T for Telegram. That acts like human 100% exactl Больше

$1000 USD за 10 дней(-я)
(15 отзывов(-а))
5.8
lightingdavid

Hello. I have good skills in "Data Scraping, Python, Scrapy, Selenium Webdriver, Web Scraping". I have working for 7+ years in this field. I 'm very interest to your project. I have checked your project description Больше

$250 USD за 3 дней(-я)
(31 отзывов(-а))
5.1
kunitsynartem

Hello! I have 2 years of experience in web scraping using Python and I'm interested in your project. I can use both Selenium and Scrapy depending on what is better for certain website. Also I can handle logins, file do Больше

$600 USD за 10 дней(-я)
(27 отзывов(-а))
5.1
smsaurabhv

‌Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHO Больше

$444 USD за 10 дней(-я)
(49 отзывов(-а))
4.9
drishinfotech

forward HI, I read your job description and would like to assist you in website scraping task. I understand your conditions and will surely provide you the code after completion of the each task. Please share Больше

$750 USD за 10 дней(-я)
(9 отзывов(-а))
4.7
albertpopov46

Dear, sir @I am fulltime freelancer@ I read your description in carefully. I am python expert and I have rich experience with scrapping. Also i have selenium experience . So I think that i can do your project in Больше

$500 USD за 10 дней(-я)
(10 отзывов(-а))
4.2
yongbeauty1996

hello how are you? I am very interested in your project. I have read your description very carefully. I can do your job in time. kind regards

$555 USD за 10 дней(-я)
(4 отзывов(-а))
4.2
NIKE9

Hi, I am a senior selenium/python expert and I can build the script as requirements in the description. I have 7+ years of professional experiences in web development. I can start immediately, also finish your proje Больше

$750 USD за 7 дней(-я)
(3 отзывов(-а))
3.6
KGeorgy

Hi, Thanks for your job posting. I've read your project description carefully. You are going to build scrapy that gets data based on keywords. As a senior scraping developer, I have rich experience in scrapy and pyt Больше

$500 USD за 10 дней(-я)
(6 отзывов(-а))
3.6
chirag9700

I have more than 6+ years of experience into IT field. Since last 6 years, I am dealing with different kind of field such like : - Laravel, CI, YII - Angular.js - Node.js - Ionic Framework - PHP - HTML - Python - Djan Больше

$666 USD за 10 дней(-я)
(2 отзывов(-а))
4.2
BoyVit85

How are you. Credit is my motto. I am expert web scraping. I can do your job with BS4 and Seleinum framework of python. I can do any project in your demand completely by my good experiences of last ago. I think thi Больше

$555 USD за 10 дней(-я)
(3 отзывов(-а))
3.1
edison4mobile

HI, how are you? I have checked your description carefully. I can say I understood fully what you want. As I have rich experienced in python(2, 3) so that your project is not problem for me. I am really confident an Больше

$777 USD за 10 дней(-я)
(1 отзыв)
2.8
vorasiddh4it

We have 11+ years of experience in software development. We have developed 400+ projects and the research paper in the field of Machine Learning, Artificial Intelligence and Image processing (GIS), Network, SEO based W Больше

$1000 USD за 10 дней(-я)
(4 отзывов(-а))
3.4