Завершено

Script to find soft 404 external pages with header response 200 (openload, powvideos,... video streaming)

The website have links to different streaming sites online like youtube, openload, powvideos,....

Generally, due to copyright claims, these videos are deleted and as a result, this links will go to an SOFT 404 error page (the header response is 200 ok).

404 - examples

[login to view URL]

[login to view URL]

[login to view URL]

I think the easiest way to eliminate at least part of these dropped links is to create a script that does the following directly on the DB, with the help of a cronjob (to avoid the server overload 1.400.000 links on the DB):

1 - the script need to open external links from my DB (table "videos" column "url")

2 - if the rezult of the external page is 404 or soft 404, then change in the DB under column "approved" from 1 to "0"

3 - It would be fantastic if it could work for all sites, but if need custom check on each external website, at least, it must work for openload, streamplay and powvideos.

Suggestion - Because these pages return a standard title for all soft 404...

For example need to create some code to first check the page title.

All 404 pages from openload , for example, will have this page title :

<title>File not found ;(</title>

Others like powvideos, will have different title based on user language (english, spanish,....)

<title>Watch </title>

<title>Ver </title>

In the case that you see it possible, it is only necessary to add a configuration file, to which I can add the titles one by one later (need to be able to open this file from FTP) ...

The answer at this question, can give some idea ... [login to view URL]

An very OLD - Python library, [login to view URL] - [login to view URL]

This is the server data:

cPanel Version 78.0 (build 23)

Apache Version 2.4.39

PHP Version 7.0.33

MySQL Version 10.1.40-MariaDB

Architecture x86_64

Operating System linux

Path to Perl /usr/bin/perl

Perl Version 5.16.3

Kernel Version 3.10.0-957.12.2.el7.x86_64

This is an small example of the table "videos" in my DB

CREATE TABLE `videos` (

`id` int(10) UNSIGNED NOT NULL,

`name` varchar(191) COLLATE utf8mb4_unicode_ci NOT NULL,

`thumbnail` varchar(191) COLLATE utf8mb4_unicode_ci DEFAULT NULL,

`url` varchar(191) COLLATE utf8mb4_unicode_ci NOT NULL,

`type` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL,

`quality` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL,

`title_id` int(10) UNSIGNED NOT NULL,

`episode_id` int(10) UNSIGNED DEFAULT NULL,

`season` int(10) UNSIGNED DEFAULT NULL,

`episode` int(10) UNSIGNED DEFAULT NULL,

`source` varchar(191) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'local',

`negative_votes` int(10) UNSIGNED NOT NULL DEFAULT '0',

`positive_votes` int(10) UNSIGNED NOT NULL DEFAULT '0',

`reports` int(10) UNSIGNED NOT NULL DEFAULT '0',

`approved` int(10) UNSIGNED NOT NULL DEFAULT '1',

`order` int(10) UNSIGNED NOT NULL DEFAULT '0',

`created_at` timestamp NULL DEFAULT NULL,

`updated_at` timestamp NULL DEFAULT NULL

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

INSERT INTO `videos` (`id`, `name`, `thumbnail`, `url`, `type`, `quality`, `title_id`, `episode_id`, `season`, `episode`, `source`, `negative_votes`, `positive_votes`, `reports`, `approved`, `order`, `created_at`, `updated_at`) VALUES

(33276636, '30-Ingles <img src=\'/[login to view URL]\'>', NULL, '[login to view URL]', 'external', 'Hd-Tv', 87982, 797195, 8, 2, 'local', 0, 0, 0, 1, 0, '2019-05-16 22:25:52', '2019-05-16 22:25:52'),

ALTER TABLE `videos`

ADD PRIMARY KEY (`id`),

ADD UNIQUE KEY `videos_url_title_id_unique` (`url`,`title_id`),

ADD KEY `videos_title_id_index` (`title_id`),

ADD KEY `videos_episode_id_index` (`episode_id`),

ADD KEY `videos_season_index` (`season`),

ADD KEY `videos_episode_index` (`episode`),

ADD KEY `videos_source_index` (`source`),

ADD KEY `videos_order_index` (`order`);

Many thanks.

Квалификация: Javascript, Linux, MySQL, PHP, Python

Показать больше html template external pages bloghoster, script read reads web pages, script flash open url pages, woocommerce cart soft 404, fehler soft 404, submitted url seems to be a soft 404, nginx soft 404, soft 404 checker, how to fix soft 404, how do i fix soft 404 errors, how to get rid of soft 404, turn pages header bar, adult video script lighttpd 404, xml response 200, redirecting external pages, adobe illustrator web pages header, external flash header, java script photoshop pdf total pages, perl script pulls information web pages, redirect script geo specific landing pages

О работодателе:
( 0 отзыв(-а, -ов) ) Belgrade, Serbia

ID проекта: #19756837

Поручен:

rupanaskar

Hello Sir/Madam, I hope you are doing well. I can make the script which will check URL via curl and check if the content has soft 404 content like 'We can't find the file you are looking for' for openload and for powv Больше

$500 USD за 4 дней(-я)
(37 отзывов(-а))
6.0

24 фрилансеров(-а) в среднем готовы выполнить эту работу за $480

fattahaabdul

I will make the script to find multiple 404 pages. I understand why the problem is coming. I have few queries . Can you answer them if you are online/ Let us discuss

$800 USD за 7 дней(-я)
(129 отзывов(-а))
8.0
NayaPakistan

Hello there. I can write the script to find soft 404 external pages and as per the requirements given in the project. Please send me a message to discuss more and get started. Thanks

$250 USD за 5 дней(-я)
(290 отзывов(-а))
6.7
Torricelli

Hi, i hope you're doing well. I've just read your project's description and i would like to help you with it. I'm an experienced web developer with the necessary skills for getting this job done. Responsibilit Больше

$400 USD за 3 дней(-я)
(202 отзывов(-а))
6.3
etuannv

Hi there, The requirements are very clear. I would approach your project using Python3. I worked a part of this project before which check Youtube video alive. I have 4+ years’ experience at Web scraping. If you'd li Больше

$555 USD за 10 дней(-я)
(70 отзывов(-а))
6.3
gongfei

Hello I read your proposal and i am very interested in your project I have rich experience in web development including php, javascript, cron job as well I will delivery great result with my best Look forward to workin Больше

$500 USD за 7 дней(-я)
(21 отзывов(-а))
6.0
Topman123

* Hi, sir. How are you ? * ABOUT YOUR PROJECT: I have similar experience with Video work. Please check [login to view URL] ABOUT ME: I just have gone through your project and I am sure I can complete your wo Больше

$500 USD за 10 дней(-я)
(26 отзывов(-а))
5.7
BestService222

Hi, How are you? I am very much interested in your project. I feel very confident about your project as I am a professional Expert, I can do your job perfectly ASAP and I want to work with you I work hard for you an Больше

$500 USD за 10 дней(-я)
(26 отзывов(-а))
5.1
whiteeagle0001

Hi Thanks for the opportunity to place bid on this project. I carefully read the comments and thought about the points to consider in the project. There are many website development and scrape experiences. I think Больше

$300 USD за 10 дней(-я)
(14 отзывов(-а))
4.2
AlexanderPGR

Hi, Dear How are you doing? I am very interested in your project. I am always ready for you. I wish you contact me as soon as possible. Let us discuss your project on chat in detail. Thanks for your regards.

$250 USD за 5 дней(-я)
(10 отзывов(-а))
4.3
AlexKolonitsky

Hello My name is Alex and I'm ready to create such script for you. I will create configuration where you can specify some template for each resource. Feel free to contact me. I have completed some web scraping jobs re Больше

$250 USD за 7 дней(-я)
(3 отзывов(-а))
3.4
razvand70

Hello, I've got over 7 years experience with python and ML. Please see my reviews. Kind regards, Razvan

$700 USD за 7 дней(-я)
(3 отзывов(-а))
3.5
rdmrla

Hello. I can help with creating the script to find the soft 404 errors. This script will read the video URL from the database and request it from the site. The response will then be parsed to check for certain k Больше

$500 USD за 5 дней(-я)
(9 отзывов(-а))
3.4
AdrianKeto

Hi I've just gone through your project requirement carefully and I'm confident to do this project. I've been developing PHP, Python and Linux with over 5 years. So I'm very interested in your project and I want to d Больше

$750 USD за 7 дней(-я)
(3 отзывов(-а))
3.0
tascoin

Hi Sir/Ma'am, I'm really interested in this job and ready to discuss with you. I have more than a decade years of experience in Python (Selenium, Django, OpenCV, Flask etc...) based Block Chain Crypto Exchange, bot, Больше

$500 USD за 7 дней(-я)
(2 отзывов(-а))
2.4
muffajalbohra53

Hello, I have gone through your requirement more carefully. Basically i have skilled in Python UI and web development and having knowledge and experience with Django/Flask web framework. I have already experienced crea Больше

$450 USD за 10 дней(-я)
(4 отзывов(-а))
2.3
eshkar

Hi, I can develop a great system to detect those soft 404s with perfect accuracy. And on top of that I can also develop a comfortable UI for you to configure new soft 404s in case a page has an unknown status (potenti Больше

$334 USD за 5 дней(-я)
(1 отзыв)
1.0
smartweb4

Hello, I am a senior linux admin with 10+ years of experience and also a php developer. I have created hundreds of scripts to automate tasks and I can help you create this script to detect 404 soft pages. I can make Больше

$400 USD за 5 дней(-я)
(0 отзывов(-а))
0.0
$555 USD за 15 дней(-я)
(0 отзывов(-а))
0.0
MedIntellego

Good Day! I am expert in scientific research, Machine Learning, Deep Learning and NLP for medical applications and would be delighted to assist you in the project of your interest in a highly professional and timely m Больше

$500 USD за 4 дней(-я)
(0 отзывов(-а))
0.0
Sureshtahiliani1

Hi, As per my proposal , we will provide a script which will have a session record as well as it will give the list of commands which has been fired by the logged in user along with it you will be able to see the outpu Больше

$611 USD за 10 дней(-я)
(0 отзывов(-а))
0.0