I need a PHP or Python script which will be run periodically to scrape the data I describe in this project and store it locally in a MySQL database running in a typical LAMP environment on a Debian 11 server. The script should start by emptying the table to be used (TRUNCATE), the populate the table only with the results of the new data scraping described in this project.
The source data is on a public county GIS mapping server and is strictly limited to the owners of property in the subdivision for which this script is being written.
Currently, we do the very tedious work of manually updating our data. To get to the data I need manually, I use these steps:
1: Open web page: [login to view URL]
2: Use the Search Field and enter "Upper Little Warm Spring Creek"
3: On the search results page, click "Select All", then click "Map Selection"
4: The resulting page is shown in the attached screenshot and the purple circles are where the data is--as explained below. PLEASE NOTE that the number of results may change from time to time, so we cannot simply use the URL that results from the search.
5: Use the "Copy to Clipboard" button on the web page to obtain the primary data.
6: Use the "Property Detail" link next to each of the property parcel entries to obtain further data to be added to the primary data.
I need this to be fully automated so that simply accessing the script will scrape and update all data in my database.
After getting to the correct data set, the results of the "Copy to Clipboard" button (circled in purple on attached screenshot) will be 23 tab delimited values for each record, from which I need the first 14 columns which need to be populated into a MySQL database table using the following column names:
Parcel, Parent, Account, TaxID, Plat, Lot, Owner, Mail_Street, Mail_CSZ, St_Addr, Deed, Location, Tax_Class, Acerage
Once the main data is imported, then every record's "Property Details" link needs to be accessed to get more data for the records already collected (2 "Property Details" links circled in purple on attached screenshot). From each "Property Details" record, I need the following additional columns for each of the records created from the "Copy to Clipboard" data:
MV_Year, MV_Total, MV_Land, MV_Improve, AV_Year, AV_Total
These values are found by searching the result from each Property Details link for the FIRST line that contains "Market Value" (MV) and the FIRST line that contains "Assessed Value" (AV) extracting the data from lines similar to this example:
2021 Market Value: $ 450,273 ($ 83,886 Land + $ 366,387 Improvements)
2021 Assessed Value: $ 42,776
results in these values:
MV_Year = 2021
MV_Total = 450273
MV_Land = 83886
AV_Year = 2021
AV_Total = 42776
As a result of both steps each line of the database table should include these columns:
Parcel, Parent, Account, TaxID, Plat, Lot, Owner, Mail_Street, Mail_CSZ, St_Addr, Deed, Location, Tax_Class, Acerage, MV_Year, MV_Total, MV_Land, MV_Improve, AV_Year, AV_Total
I am available to explain further as needed and to participate in testing.
Please do NOT ask me for access to my server as this script needs to run in most any current Linux-based LAMP environment
1) MySQL "CREATE TABLE" script to set up the database table I describe in this project.
2) PHP or Python script to accomplish the tasks described in this project.
Great performance on this project will likely lead to further projects from me as well.