How to merge the different lines of files?

What is the fastest command line way to merge the different lines of files? For example, I have two files:

a.txt:

foo bar foobar

b.txt

foo foobar line by bar

And I would like to get the following output:

foo bar foobar line by

Is there any fast way to merge files like the example above? (The order of the lines isn't important)

3 Answers
3

Use awk seen if you don't want to sort the file:

awk seen

$ awk '!seen[$0]++' a.txt b.txt foo bar foobar line by

Does it delete the duplicated lines in a single file?
– kbenda
Aug 31 at 11:26

@KrisztiánBenda, it just print the uniq lines in each file.
– msp9011
Aug 31 at 11:33

@KrisztiánBenda you can redirect the out to another file, if required.
– msp9011
Aug 31 at 11:34

Thanks @SivaPrasath for your comment. It seems to me seen is not a valid command in my environment. And apt-get install seen does not find it.
– kbenda
Aug 31 at 11:41

apt-get install seen

@KrisztiánBenda It's an associative array in awk. You may use any array name and seen is often used for these kinds of operations. awk '!p[$0]++' would work just as well. Note that this solution is appropriate for reasonably large amounts of data, but that memory consumption grows with the number of unique lines in the input.
– Kusalananda
Aug 31 at 12:20

awk

seen

awk '!p[$0]++'

$ sort -u a.txt b.txt bar by foo foobar line

This would sort the (combined) contents of the two files, removing the duplicate lines. The downside is that if a line is duplicated in one of the files, this would also be removed.

To write the result to c.txt, use

c.txt

sort -u -o c.txt a.txt b.txt

or a standard redirection

sort -u a.txt b.txt >c.txt

Fastest/shortest is perhaps sort -u a,b,-oc.txt then ?
– steve
Aug 31 at 10:51

sort -u a,b,-oc.txt

@steve Gah! I would avoid that. That requires a shell that understands brace expansions, and a sort implementation that allows for options to come after the operands on the command line. It's far safer to just be explicit, at least with the command line options and their arguments. You may use sort -u -o c.txt [ab].txt if you want.
– Kusalananda
Aug 31 at 10:56

sort

sort -u -o c.txt [ab].txt

Based on your desired output, I presume that you just want both files merged with only unique strings.

In which case, cat, sort and uniq can do this for you:

cat

sort

uniq

cat a.txt b.txt | sort | uniq > c.txt

cat opens the contents of both files

cat

sort sorts the output alphabetically

sort

uniq lists only unique strings

uniq

> c.txt puts all of the output in a new file c.txt

> c.txt

c.txt

sort command is more enough to do..
– shas
Aug 31 at 12:10

so I have learned.. that being said the answer is still an alternative, albeit uneccessarily long-winded (relatively anyway) working answer.
– RobotJohnny
Aug 31 at 12:15

this is also a fast method.
– prosti
Aug 31 at 22:54

Thanks for contributing an answer to Unix & Linux Stack Exchange!

But avoid …

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

But avoid …

To learn more, see our tips on writing great answers.

Required, but never shown

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

搜尋此網誌

Dfyjkt