#20: Multi-line sed search and replace
sed is a very famous stream editor on UNIX systems. It is very powerful and versatile and makes manipulation of streams and text files very easy. However, it has a steep learning curve. Like vi/vim, sed might seem very bulky at the beginning, but as soon as you begin to understand the tool, it makes your workflow very efficient.
sed is built to process strings (either from STDIN or from a file) line by line. Therefore, you can't search for multiple lines in a way like this:
sed 's/foo\nbar/bla\nblub/'That wouldn't work because the pattern space, in which all operations are performed, only contains one line at a time. Also replacing newline characters wouldn't work that way because they are stripped in pattern space. But there are several ways to work around this. I'll be showing you three ways of performing multi-line replacements.
The first way is that one, you normally find when googling for that topic. It makes use of the N command, which reads the next line and appends it to the pattern space, separated by a newline character.
sed '/foo/{ N; s/foo\nbar/bla\nblub/ }'This looks for foo and, if found, appends the next line and does the replacement. But this method has a catch. If you have an input string like this, this won't work
foo foo bar
The first occurrence of foo is found and the next line is appended to pattern space. But of course
foo foo
doesn't match the multi-line pattern, so the pattern space is replaced by the next line, which is
bar
but this doesn't match foo. You see, this method is very rough and does only work in a few cases.
But we can optimize this. Besides pattern space, sed also provides the so called hold space. This is just a temporary buffer on which no operations are performed. We can use this to read the whole input into it first and then replace the pattern space with the contents from hold space. That looks like this:
sed -n '1h; 1!H; ${ g; s/foo\nbar/bla\nblub/ p }'That first reads the first line from pattern space into hold space (1h) replacing all contents which currently exist in hold space. Then all lines except line 1 are appended to hold space (1!H). The reason why we cannot only use 1,$H is that this would result in a blank line at the beginning since hold space has not been emptied. As soon as the end of the string is reached (range marker $), a subclause is opened which writes contents from hold space into pattern space (g) and does the replacement. Because we have read everything into hold space and then into pattern space, we would get double output. To avoid this, the parameter -n (no output) is set and the edited final string is printed manually with the p command from within the subclause. This method works remarkably well, but you should note that it is much slower if the stream/file is very long. One advantage of sed over many other tools is that it reads line by line, so it doesn't take more memory when working on long strings. This advantage is abrogated with this method. Keep that in mind.
Another way that came to my mind is to omit the hold space and read everything directly into pattern space. That's a mixture of method one and two.
sed '1!N; s/foo\nbar/bla\nblub/'sed automatically reads line 1 into pattern space, so we only have to append all the others. We do this with 1!N, which appends all lines except the first one to pattern space. Then the replacement is performed. Done! Short and nifty. The only problem with this method is that is has problems with multiple replacements (g flag). For this better use the second method. Of course, you can also use this with the -n parameter and p command, but then you have to set a semicolon after the replacement command, otherwise you'd only get the parts which have been replaced. The rest of the string would not be printed to the screen. So
sed -n '1!N; s/foo\nbar/bla\nblub/ p'is something different than
sed -n '1!N; s/foo\nbar/bla\nblub/; p'The first one would only output
bla blub
and the second one
foo bla blub
That's it, hope you learned a bit.
Read more about sed multi-line replacement:
- GNU.org: sed info manual
- Unix Sed Tutorial: Multi-Line File Operation
- sed and Multi-Line Search and Replace
Trackbacks
remember: Multi-line sed search and replace → https://t.co/89mvNpDmEL
Comments
This was really informative but I still havent been able to get it to work. What is your input on the command? A full working example would really make this article fantastic. Thanks
Hi Jeff,
you can either specify an input file like so:
bc. sed ‘pattern’ input-file
or pipe the output through STDIN:
bc. echo foobar | sed ‘pattern’
Hi, thanks for this topic!!! I read it, but I am still to "stupid" two replace the second of two lines
In my example file I got those two lines: <param-name>Type</param-name>^M <param-value>green</param-value>^M
and I want to replace it with <param-name>Type</param-name>^M <param-value>red</param-value>^M
My code looks like this: REPLACELINE1="<param-name>Type<\/param-name>^M" REPLACELINE2="<param-value>green" REPLACEWITHLINE2="<param-value>red"
sed -n "1!N; s/$REPLACELINE1\n$REPLACELINE2/$REPLACEWITHLINE2/; p" "$FILE" > "$FILE.NEU"
Do you know what I did wrong?
I would appreciate any help!!
/Virginia
Hi Virginia,
you might want to try this:
REPLACELINE1=’<param-name>Type<\/param-name>^M’ REPLACELINE2=’<param-value>green’ REPLACEWITHLINE2=’<param-value>red’
sed -n "1h; 1!H; \${ g; s/$REPLACELINE1\n$REPLACELINE2/$REPLACEWITHLINE2/ p }" "$FILE" > "$FILE.NEU"
(be careful when doing copy&paste as the blog software replaces the string delimiters/quotation symbols in the comments with their real typographic counterparts)
Hi…this is Raj..i am new to shell scripting ..i have to write a script to search and replace multiple lines in a file.. Example .
cat myfile.txt
-a myown car dkdjdj / ghee 500>= / -w mdjdfdj -S jfdjf -F nfjfjf . . .etc..
i have to search multiple lines and if not found replace that line in afile. I used below command but not works for me
sed -i ‘s/-a djdjdjdj -s nff -f 500> = / /-a djdjdjdj -s nff -f 500> =//g’ myfile.txt
please suggest me how to do it in better way. Thanks in advace.. SPR
Hi,
I don’t really know what you want to replace since your pattern doesn’t match the contents of the file file (there is no "-a djdjdjdj" in the file, only "-a myown car dkdjdj"). What I can say for sure, though, is that you have a syntax error because you either need to use different delimiters (e.g. | instead of /) or escape every occurrence of / in the search and replace string.
Thank you very much, this is a very clear explanation. For the first time in years, I could use sed instead of awk for simple cases. What follows could help me to automate updates on our DNS server’s config files. sed -e ‘/^@[ ]IN[ ]SOA/N;s/^(@.\n[^0-9])[0-9]*/\1’$(date +"%Y%m%d%H")’/’
Hi Janek,
I need to replace <tag key="student1" value="FALSE" />
with <tag key="student1" value="TRUE">
The new line character and tab spaces in the first search string causes the issues. Can you please guide me to replace it?
You could try it like this:
pre. sed -n ‘1h; 1!H; ${ g; s/(<tag[[:space:]]+key="student1"[[:space:]]+value=")FALSE("[[:space:]]*\/>)/\1TRUE\2/g; p }’
(ignore the double escaped HTML tags, they should actually be < and >, not < and >)
However, I don’t think, sed is the right tool for you. Replacing HTML or XML with regular expressions isn’t really a thing (XML is not a regular language). For a few quick ad-hoc fixes, it might work, but if you really need to do some more (and guaranteed to be valid) changes, you should really use an XML parser. Remember, that the regular expression also won’t match if the attributes are in the wrong order, the tag is not self-closing etc.
How do I print a file, I want to first search for StartPattern1 and EndPattern1 with Matchpattern1 and print from startpattern1 and Endpattern1. Then I want to do same startPattern1 and Endpattern1 this time with Matchpattern2. i was able to accomplish this by below
sed -n ‘/interface dsl/h;/interface dsl/!H; /!/ {x;/dsl profile INTRO/p;}’ /home/ProfileChangeSelect/FILE
now i want to add " dsl profile VALUE" in addition to dsl profile INTRO
What is the syntax? THanks
You might want to consider splitting it up into multiple commands and pipe them together.
RT @reflinux: #Advent series "24 Short #Linux #Hints", day 20: Multi-line sed search and replace http://bit.ly/g53KTc