Create a Fast Web Crawler to Extract Google SERP Data


I need someone to develop me a fast and powerful web crawler to extract data from Google SERP.

Important key features;

1. Must have a user interface protected by user and password where I can manage my campaigns.

2. Must have rotating proxies support

3. Multi threading support

I want a dashboard, where I can create/edit/delete my campaigns. For every campaign, I will set a google query (ie: intitle:"fashion" "blog") and decide how many results I want to extract.

When I will click on the "Start Crawler" button, I want to see the results in real time.. so If I decided to crawl the first 100 000 results for this query, if the crawler have extracted 1000 results, it should show 1000 results extracted out of 100 000.

Once the extraction is completed, I want to receive an email that notify me that the campaign has finished.

Once the process is completed, I want to have 3 options;

1. Remove duplicated domains (Remove all entries that is for the same domain) ie:

[login to view URL]

[login to view URL]

[login to view URL]

Should keep just first entry so "[login to view URL]" and delete all the other entries.

2. Update metrics (This will use the SEMrush API to update the keywords / organic traffic for every extracted domains)

3. Export the data in Excel format. (Columns; Full URL / SEMrush Keywords / SEMrush Organic Traffic)

Навыки: Веб-скрейпинг, PHP, Python, JavaScript, Архитектура ПО

Показать больше: extract google analytics data limit, php extract google serp, extract google analytics data, web crawler extract description php mysql, extract google analytics data xml, fast web crawler mysql, web crawler extract data, java web crawler extract web page title, extract google analytics data daily, web crawler extract data website, web crawler extract phone number, web crawler extract data excel, extract google play data, web crawler script google, code web crawler extract data website, web crawler extract information java, program web crawler extract information website, create simple web-crawler and email sender, how to create a web crawler

О работодателе:
( 44 отзыв(-а, -ов) ) Sabrevois, Canada

ID проекта: #23957254

15 фрилансеров(-а) готовы выполнить эту работу в среднем за $596


Hey there, I have 4 years of working experience in web technologies and I have a strong working knowledge of every possible framework which is in the market. In terms of client-end I know: Javascript, PHP, Python, Sof Больше

$1250 USD за 29 дней(-я)
(110 отзывов(-а))

Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this project ! I can start immediately and finish it within the Больше

$500 USD за 7 дней(-я)
(180 отзывов(-а))

Hi. I did read the project description and have a few questions. 1. Do you need the script as well or data only? 2. What is the format of the output data? CSV is OK? We can do other formats as well. 3. Which fields Больше

$500 USD за 4 дней(-я)
(151 отзывов(-а))

Hi there, Being an expert in Python and matlab programming I can do this project for you. please leave a message on my chat so we can discuss the budget and deadline of the project. Also, have a look at my profile for Больше

$750 USD за 3 дней(-я)
(32 отзывов(-а))

Hi there, Greeting!! I have more than 10 years of experience in Web & App development and multiple technologies. I am committed to providing you high quality services. My goal is to satisfy clients by providing high q Больше

$750 USD за 7 дней(-я)
(48 отзывов(-а))

hello sir , I am a full time freelancer and have 10 years of experience in web and window development in asp.net c#, javascript,sql server . and Android mobile development i can do work as per your requirement . Больше

$500 USD за 7 дней(-я)
(9 отзывов(-а))

Hello I'm familiar Page Scrapping with Phantom JS and php script. I have some examples. I read your description and i can start right now. Please discuss details on chat, Thanks

$700 USD за 7 дней(-я)
(11 отзывов(-а))

Hi there, After reading your offer, this looks like a perfect fit for my skill sets so May I discuss with you for further details about this project? I'm a web developer with over 7 years experience and very strong i Больше

$250 USD за 4 дней(-я)
(8 отзывов(-а))

Hello Dear, Yes, I can work for you as per your project description with amazing exploration and experimental research abilities of mine organization is perfect for that you may require. My work fulfills as well as ke Больше

$600 USD за 10 дней(-я)
(8 отзывов(-а))

Hello, I checked out your requirements.I have 5+ years of working experience and Expertise in IOS , Android , Web Development, PHP, JavaScript, NodeJS, HTML,C++ ,Wordpress, CSS, RestAPI, Third-Party integration. I a Больше

$500 USD за 7 дней(-я)
(2 отзывов(-а))

Hi there. I have carefully read your job description with great interests. I have experience with python, django,selenium and bs4(beautifulSoup) for about 5 years. Please visit my past references here. -http://cars Больше

$500 USD за 7 дней(-я)
(4 отзывов(-а))

Hi there! I have read your description carefully and I am confident I can full fill your requirement. I provide my skills in hopes of working with you. I am a web scraper and have 7+years of ample experienced in web s Больше

$500 USD за 7 дней(-я)
(3 отзывов(-а))

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spide Больше

$500 USD за 7 дней(-я)
(0 отзывов(-а))

Hi I have read your post content carefully and can develop your python project as my experience. That's why I have rich experience with Python. I am a new freelancer on freelancer.com, but I can provide you robust and Больше

$500 USD за 7 дней(-я)
(0 отзывов(-а))

Nice to meet you Hi I am automation expert If there is any website you need to scrape data from (no matter the format, what blocking tools there are). I can deliver the data. If you do the same thing on the internet Больше

$637 USD за 9 дней(-я)
(0 отзывов(-а))