GoGreen PC Tune-Up™
Learn More
Insta-Install™
this is how ssl encrypt our websites
MTNCOMP | List View | Table View | myBlog (1762 Entries)
myBlog Home

Blog

Linux Bash reading a file not ending with EOF without blank line

Linux Tux

by Mountain Computers Inc., Publication Date: Tuesday, June 2, 2020

View Count: 1327, Keywords: Linux, Bash, Read File, EOF, Blank Line, JSON to CSV, Comma Separated, Hashtags: #Linux #Bash #ReadFile #EOF #BlankLine #JSONtoCSV #CommaSeparated


If you are processing a file with 188 million JSON records and the normal script used to parse that file never ends, its because the last line of the JSON file does not have a blank line.
 
Are there remedies, yes. for me, I just concatenated a \n to the end of the file and the script i used to parse the records worked fine.
 
yes, i have checked out the jsontocsv.php options and many more. sometimes, you just want to brute force the process to see if and how long it takes and works. 
 
f.sh

#!/bin/bash
input="/home/andyf/web/dns/dns3.json"
while IFS= read -r line
do
    # echo "$line"
    #echo ${line/\{\"timestamp\":\"/}
    t=${line/\{\"timestamp\":\"/}
    u=${t/\",\"name\":\"/,}
    v=${u/\",\"type\":\"cname\",\"value\":\"/,}
    echo ${v/\"\}/} >> dns3.out
done < "$input"
 
sample file, tail from my domain records source for processing domains. source file size 20GB, output file size 11GB.
 
** Ultimate destination is MariaDB into a dnsDB schema where it can be processed via curl or some version of it with overrides.
 
(base) andyf@MTN-SYS:~/web/dns$ tail dns3.json
{"timestamp":"1585945700","name":"zkzcn.edu.tianmeidao.com","type":"cname","value":"s1.hourxo.com"}
{"timestamp":"1585945571","name":"zkzcpv.com","type":"cname","value":"nmz188.us16.iisok.net"}
{"timestamp":"1585945648","name":"zkzcv.sohu.hg6386.com","type":"cname","value":"park.jinmi.com"}
{"timestamp":"1585945543","name":"zkzcvjyjflff.weebly.com","type":"cname","value":"pages-wildcard.weebly.com"}
{"timestamp":"1585945568","name":"zkzcwtep.btssr.com","type":"cname","value":"btssr.com"}
{"timestamp":"1585945521","name":"zkzcwyh.com","type":"cname","value":"374945.vhost621.cloudvhost.cn"}
{"timestamp":"1585945558","name":"zkzcy.vrphoria.com","type":"cname","value":"westexpired.dopa.com"}
{"timestamp":"1585945501","name":"zkzczs.com","type":"cname","value":"host3123841.xincache1.cn"}
{"timestamp":"1585945531","name":"zkzd.com","type":"cname","value":"overdue.aliyun.com"}

(base) andyf@MTN-SYS:~/web/dns$ tail dns3.out
1585945700,zkzcn.edu.tianmeidao.com,s1.hourxo.com
1585945571,zkzcpv.com,nmz188.us16.iisok.net
1585945648,zkzcv.sohu.hg6386.com,park.jinmi.com
1585945543,zkzcvjyjflff.weebly.com,pages-wildcard.weebly.com
1585945568,zkzcwtep.btssr.com,btssr.com
1585945521,zkzcwyh.com,374945.vhost621.cloudvhost.cn
1585945558,zkzcy.vrphoria.com,westexpired.dopa.com
1585945501,zkzczs.com,host3123841.xincache1.cn
1585945531,zkzd.com,overdue.aliyun.com
 
references:
https://www.cyberciti.biz/faq/unix-howto-read-line-by-line-from-file/
 https://stackoverflow.com/questions/10929453/read-a-file-line-by-line-assigning-the-value-to-a-variable
 
more to come...
if you found this article helpful, consider contributing $10, 20 an Andrew Jackson or so..to the author. more authors coming soon
FYI we use paypal or patreon, patreon has 3x the transaction fees, so we don't, not yet.

© 2024 myBlog™ v1.1 All rights reserved. We count views as reads, so let's not over think it.