Title: Unix replace new lines inside quotes
Tags: regex;bash;unix;awk;sed
Question: I am trying to figure out the below in Unix.
I have a csv file which is separated into columns and some of the columns have some text with new line characters. Usually the text starts and end with quotes. I would like to eliminate those new line characters between the quotes so that the text appears in one line.
For example the content looks like below:
HEADER1, HEADER2, HEADER3, HEADER4
data1, data2, data3, "Text here with
new line
characters"
And I would like it to be displayed like below:
HEADER1, HEADER2, HEADER3, HEADER4
data1, data2, data3, "Text here with new line characters"
I tried to use tools like awk and sed but without much luck.
Appreciate any help
Comment: Can your quoted text contain commas? Can it contain escaped quotes as either `\"` or `""`?

Comment: Possible duplicate of [bash replace newlines into quotes](http://stackoverflow.com/questions/16036744/bash-replace-newlines-into-quotes)

Comment: @Mr.Llama That looks close but I tried several of the answers in that question including the accepted answer.  None worked for the OP's file.  They removed _all_ newlines and the OP only wants to remove quoted newlines.

Here is the accepted answer: With GNU awk for multi-char RS and
1) the 3rd arg to match():
$ awk -v RS='^$' -v ORS= '{while ( match($0,/"[^"]+"/,a) ) {gsub(/\n/," ",a[0]); print substr($0,1,RSTART-1) a[0]; $0=substr($0,RSTART+RLENGTH)} print}' file
HEADER1, HEADER2, HEADER3, HEADER4
data1, data2, data3, "Text here with new line characters"
2) the 4th arg to split():
$ awk -v RS='^$' -v ORS= '{n=split($0,f,/"[^"]+"/,s); for (i=0; i<=n; i++) {gsub(/\n/," ",s[i]); print f[i] s[i]} }' file
HEADER1, HEADER2, HEADER3, HEADER4
data1, data2, data3, "Text here with new line characters"

Here is another answer: $ sed -r 'H;1h;$!d;x; :a; s/(, *"[^"]*)\n/\1 /; ta' file.csv
HEADER1, HEADER2, HEADER3, HEADER4
data1, data2, data3, "Text here with new line  characters"
The above looks for any occurrence of comma, followed by zero or more spaces, followed by one double quote, followed by anything except a double-quote, followed by a newline.  If found, it removes the newline.  This is repeated as long as such lines are found.
The above was tested under GNU sed.  For BSD sed, try:
sed -E -e 'H;1h;$!d;x' -e ':a' -e 's/(, *"[^"]*)\n/\1 /' -e 'ta' file

Here is another answer: The ORS can be toggled when the quotes within a line are unbalanced, replacing new-lines within the quoted text.  It will not work with C-style escaped quotes \", but the normal style for escaping quotes in CSV files keeps them balanced "" (RFC4180 2.7).
awk -F'"' 'NF&amp;&amp;NF%2==0{ORS=ORS==RS?" ":RS}1' file.csv

Here is another answer: With perl :
perl -pe 's/\n/ / if $. &gt;= 2' file
With awk :
awk 'NR &gt;= 2{printf("%s ", $0);next} 1' file

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 4, 'CommentCount': 3, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2015:00:10/08/15', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<regex><bash><unix><awk><sed>', 'Title': 'Unix replace new lines inside quotes', 'ViewCount': 2689, 'id': 33026137, 'user_AboutMe': '', 'user_AccountId': 1915780.0, 'user_DisplayName': 'nikkatsa', 'user_Reputation': 1701.0, 'user_Views': 164.0, 'user_id': 1728402.0}}