I am seeking a seasoned and savvy WEB SCRAPE GENIUS to build/modify a generic web scrape application!!
I hired another developer to build a .net web scrape application a couple of years ago. Now, I want to make more modifications to the application.
I have prepared a requirements document that explains how the application should work and I have provided pseudo-logic for the overall processing model of the web scrape application.
The goal is to build an application that could model and process 100s or even 1000s of different web scrape jobs using the HTTP / HTTPS model -- not a user simulation / automation model. Key features include --
a. automated login to websites and setting cookies
b. proxy server usage
c. get and post automation
e. xml, database, and file/csv file saving options
f. configuration file witih over 25 web scrape logic control settings
g. separate job scheduler to triggering scrape jobs to run
The xml configuration file is tailored to fit the website being scraped.
I would like to retain someone I could work with on an ONGOING basis. A monthly retainer quote would be valuable.
The idea is the spend the first 4 weeks building the application. I would provide the current code base in .NET. However, I am considering converting the entire app to PHP as it is cheaper to host linux/php machines than windows vps machines.
Then we would test and modify the application on several web scrape jobs. Most of my web scrape jobs are millions of rows. So architecture and multi-threading are essential.
Start November 1, 2010