Закрыт

scrape google api return

Language: PHP, NO FRAMEWORKS

Database: mysql

API used: [login to view URL]

Table A (named rfpsites_mainlist)

name varchar

state (varchar) 2 letter us state code

type varchar(city, county, school district, university, college)

Table B (named serpurl)

id (int auto increment)

name varchar

url

state (varchar) 2 letter us state code

type varchar(city, county, school district, university, college)

createddate (auto populate on insert)

key

dataforseo_task_id

runid

Table C (named dataforseoloc)

import of csv file from [login to view URL]

Testing:

You must be able to run without arror against 100 random test search queries of your choice, twice. Simply populate your test search queries into the name column on table A, as well as two letter US state names in the state column in table B. Half way through your run of 100, you will do a ctrl-C to force close the process, and rerun the script. It must only retrieve the remaining portion from where it left off. Once that is complete, do one additional run of the script against the complete 100 records. You should have 10,000 - 15,000 rows in table B at this point.

The script will:

1

Determine the current run id. This will either allow a full run, or a partial run to complete, if it was interrupted. Look at table B for the highest runid value (select runid from tableB order by runid DESC limit 1). Loop through all of the concatted ("[login to view URL] [login to view URL] [login to view URL]") in table A, to see if that value exists in table [login to view URL] with the same runid value. If they all exist, increment the runid upward one, and start an entire new run. If they partially exist, skip past any key values that exist in both tables for that runid, and complete the data being gathered into table B. Each key will need to be in table B a minimum of one time, and a maximum of 100 for the given runid.

If there are no records in table B, set the runid to 1.

For the concatted values being skipped, skip that one and check the next match.

For each of the concatted values that are not being skipped, proceed with step 2

2

identify one of the location id to use for the query

For entries in table A with a state value, convert the 2 digit state to full name, and prepare a query as follow:

"Select loc_id from tableC where loc_name_canonical like "*StateName, United States" and loc_country_iso_code limit 1"

For entries in table A without a state value, we will just leave loc_id undefined

3

for each for of the records in tableA, run a query against the serp api from dataforseo

do a multi_curl call to the url (enough to try and saturate 100 megabit)

set key to the value of these fields concatted together with a space in between "[login to view URL] [login to view URL] [login to view URL]"

set loc_id to the one retrieved in step 2

set se_name to "[login to view URL]"

4

parse the results and keep track of which key resulted in which multi_curl return

skip the paid results

skip any url with these words in the domain name - "weather", "twitter", "glassdoor, "census", "zillow", [login to view URL], mapquest, [login to view URL], [login to view URL], casetext, or "google" in it

5

for each of the results remaining from step 4, check to make sure a record does not already exist in tableB for that url. you can consider it a match if it is simply http vs https, but the frst of the url is identical. the domain portion is not case sensitive, but the path is case sensitive. If it does already exist, you can skip the insert.

6

insert a record into table B, with :

populate [login to view URL] to concat ([login to view URL], [login to view URL], [login to view URL])

url = from dataforsea return

dataforseo_task_id = from dataforsea return

runid = genereted in step 1

7

repeat steps 5-6 for every url returned from each api call, not skipped in step 4-5

8

repeat steps 2-7

Квалификация: PHP, MySQL, Веб-скрейпинг, cURL

Показать больше scraper api javascript, web scraping, perfect scraper api, google web scraper, google url scraper python, scraper api, email scraping api, scrape google search results python, set google api translate default language drop, google api save page scrape search results, google api gps data, google api save xml data, google api data extract feet, api scrape google search results, scrape google finance historical data html, google api language translate, scrape google products data, google api marker custom data, google dictionary api target language, google api json data mysql

О работодателе:
( 21 отзыв(-а, -ов) ) Kennesaw, United States

ID проекта: #20157634

11 фрилансеров(-а) в среднем готовы выполнить эту работу за $214

minhhoangduc

I'm a PHP developer and having over 10 years in programming. I can work with Bootstrap, HTML, CSS, Javascript, jQuery, System admin + API integration as well. Please take a look at my latest projects on https://www.f Больше

$250 USD за 4 дней(-я)
(629 отзывов(-а))
8.3
xtremewebtech

Hey Preferred Freelancer here! We are placing our bid so we can discuss the aspects in detail and we would love to help you with your project "scrape google api return". We are expert in PHP, Web Scraping, MySQL, cUR Больше

$250 USD за 4 дней(-я)
(287 отзывов(-а))
8.3
idragon712

Dear Client. As a senior web developer, my skill is perfect in your job. I’m very confident in building resource management tool so I wanna dive in to this task right now. ---Main skills--- -Backend PHP : Laravel, Cod Больше

$140 USD за 7 дней(-я)
(79 отзывов(-а))
7.6
ferozstk

Hello, After reading your project details I believe I'm suitable for this project. As I'm expert on it with more than 7 years experience. Please feel free to contact me. I am looking forward to hear from you. Больше

$100 USD за 4 дней(-я)
(142 отзывов(-а))
6.7
techwelfare

Just give us the chance to prove ourselves....... We have some questions to clarify,please reply us so we could discuss the project in details. Thanks & Regards Moumita

$232 USD за 14 дней(-я)
(173 отзывов(-а))
7.2
adampohp79

Hello, I read out your task completely and I am interested in your post. I have confident that I will provide you quality work within your deadline. I have a deep passion for research and I guarantee that all of my wo Больше

$140 USD за 7 дней(-я)
(59 отзывов(-а))
6.2
leemilun

Hi, nice to meet you and I hope you are always going well. Firstly I notify i have not used bidding with random or auto. I have checked your project carefully, so that i feel I can do this project. I am good in web bas Больше

$200 USD за 3 дней(-я)
(76 отзывов(-а))
6.7
eden134

I have read your description very carefully. If you keep in touch me, I can help you greatly. So I have a good team in many experiences in Web developing. Any project is a piece of cake to us! Trust us! If you keep in Больше

$150 USD за 7 дней(-я)
(4 отзывов(-а))
3.8
vikasp007

I've 7+ years of experience in Web Development/Web design, Mobile App Development(IOS/Android), Graphic design, Logo Design Business Cards & Stationery, Flyers & Brochures, Packaging Design, Web & Mobile Design, Social Больше

$140 USD за 7 дней(-я)
(2 отзывов(-а))
2.5
limondteam

"Hi, How are you? I saw your description carefully and I am confident I can full fill your requirement. So I provide my skills in hopes of working with you. I am full-stack developer and I have 2+ years of ample experi Больше

$250 USD за 2 дней(-я)
(4 отзывов(-а))
2.6
kolya113

Hello, I am so curious about your project. I have 8 years of web development experience. While this tenure, I have managed so many database administration so that I can fully handle your SQL database queries and admi Больше

$500 USD за 7 дней(-я)
(1 отзыв)
1.1