Re: [gentoo-user] Re: [OT] Question about duplicate lines in file

[Alan McKinnon <alan@linuxholdings.co.za>]

On Monday 12 June 2006 19:55, Christer Ekholm wrote:
> Teresa and Dale <teendale@vista-express.com> writes:
> > Thanks, read the man page, it was short so it didn't take long. 
> > I tried this:
> >
> > uniq -u /home/dale/Desktop/hosts /home/dale/Desktop/hostsort
> >
> > It doesn't look like it did anything but copy the same thing
> > over. There are only 2 lines missing.  Does spaces count?  Some
> > put in a lot of spaces between the localhost and the web address.
> >  Maybe that has a affect??
>
> The problem with uniq is that it (according to the manpage),
>
>   "Discard all but one of successive identical lines"
>
> You need to have a sorted file for uniq to do what you want, or
> sort it with the -u  option
>
>   sort -u hosts > hostsort
>
> If you don't want to ruin your original order you have to do
> something else. This is one way of doing it with perl.
>
>   perl -ne 'print unless exists $h{$_}; $h{$_} = 1' hosts >
> hostsort


Almost there :-)

If /etc/hosts has these lines:
127.0.0.1 localhost
127.0.0.1  localhost
uniq will see these as different even though they are actually the 
same entry. So he needs something like tr to squash spaces. This will 
do it (as root):

cat /etc/hosts | tr -s ' ' | sort | uniq -i > /etc/hosts.new

If the new file is OK, use it to overwrite /etc/hosts

Explanation so Dale knows what I'm asking him to do:
cat send the file to tr
tr finds all cases of two or more consecutive spaces and replaces them 
with one space
sort does a sort
uniq finds consecutive lines that are the same and throws away the 
extra ones. The -i is there just in case two entries differ in case 
only (as FQDNs are strictly speaking case insensitive). As mentioned 
by others, uniq only matches consecutive dupes, so the list must be 
sorted first
> /etc/hosts.new writes the final output to the named disk file

Cheers,
alan

p.s. Those 15,000 entries in your hosts file are, um, a lot :-)


-- 
If only me, you and dead people understand hex, 
how many people understand hex?

Alan McKinnon
alan at linuxholdings dot co dot za
+27 82, double three seven, one nine three five
-- 
gentoo-user@gentoo.org mailing list