Bash script to read text values in files recursively and prepare a new file ready for MySQL

We have a series of text files which contain lists of log file information in this format ->



There is no delimitation around the fields other than the full colon ":" separating the 2 values and obviously a new line at the end of the 2nd value. There are however some errors in these files, some of the lines don't follow the correct format above and we need these removing otherwise we wont be able to use LOAD DATA into MySQL when we come to import them.

These files are stored in multiple nested directories, some of which are 5 or 6 levels deep and each directory can contain between 50 and 200 text files.

We are looking for someone to write a simple BASH script to run on ubuntu 16.04 in the root directory of all these files. The script will traverse its way down each file and directory and cleanse the data. The script will need to open each file, read it one line at a time and check that each line conforms to the above format. If it is in the correct format, add it to a new "cleansed data" text file in the same (value1:value2) format in the current directory being checked. When its finished reading all the files in that current directory, it will save the cleansed data text file and move onto the next dorectory.

We have 98 directories of this data and hundreds of files within them. The output of the script will be 98 "cleansed" text files, one in each directory containing all the correctly formatted lines from all the files in that directory.

We'll then use each new cleansed data file with the LOAD DATA facility to upload the new value1:value2 checked pairs.

Квалификация: Linux, MySQL, Shell Script

Показать больше shell script to read a file line by line and write to another file, how to read a file line by line in unix using for loop, shell script to read a file line by line in unix, bash while read, shell script to read data from text file, bash read file line by line, bash read first line of file, bash script read file into variable, script unzip files recursively, script read files, script read text pdf, script read text file, bash script read logs, bash script read columns spreadsheet, script read text sentence, vba script read text file, script read text files, vbs script read text file export html file, lotus script read text files, script find text xml files

О работодателе:
( 1 отзыв ) Lincoln, United Kingdom

ID проекта: #18159808



HI, I have ten years commercial Linux experience as a systems administrator. I can write a script to filter your files. Hope to speak with you soon.

£23 GBP за 1 день
(14 отзывов(-а))

11 фрилансеров(-а) в среднем готовы выполнить эту работу за £57


Bash is one of my best skills. I use it every day. All I would need is a few sample files, and some nodes about data types in the file.

£20 GBP за 1 день
(245 отзывов(-а))

I have worked as server admin for big names like [login to view URL] and servercentre.net. Over 8yrs of working with cpanel, directadmin and plesk servers. Can help you with this. Let me know.

£50 GBP за 1 день
(124 отзывов(-а))

Hello I can do it immediately in python Thanks ...............................................................

£111 GBP за 1 день
(2 отзывов(-а))

Hello Sir, I am agile safe 4.0 certified I have 7 year of experience in Software development including technologies java, j2ee, spring , hibernate, ,Rest apis, php unix shell scripts, apache tomcat, jboss, server co Больше

£10 GBP за 1 день
(3 отзывов(-а))

Hi, there. I am happy to apply in your project. Your project is interesting to me. An experienced Bash/Shell script expert here. If you want to review me, you can go through my feedbacks and git repos. https://git Больше

£222 GBP за 1 день
(7 отзывов(-а))

Hi, I can write a python script for this that will full fill the purpose. As you will require data cleanse, so I feel no perfect script can do. There will always scenarios that are not present in current data but can b Больше

£33 GBP за 2 дней(-я)
(3 отзывов(-а))

Hello, I'm interested in the project. Right now I have a small web development company with 2-3 web developers. We are currently working on the back-end of our website and when we fully launch our front-end will be com Больше

£18 GBP за 1 день
(0 отзывов(-а))

Hi, I wrote many such nifty content aware scripts. This is doable with shellscript, ruby, perl and almost any language. I will choose the one that will get the job done best. I hope you decide for me and be amazed!

£55 GBP за 1 день
(0 отзывов(-а))

6 Years of Experience, Expert in all kind of applications, Laravel 5.1, 5.5, Spring Mysql, SQL Server.

£23 GBP за 2 дней(-я)
(0 отзывов(-а))

As your description of what the script should do sounds quite complete, as I understand it, the script should: - take as a parameter the root directory of where the log files are - recursively traverse the entire tree Больше

£61 GBP за 2 дней(-я)
(0 отзывов(-а))