[gentoo-user] Installing outside of Portage & cruft removal

[gentoo-user] Installing outside of Portage & cruft removal

[Grant]

I'd like to install the latest miro from their nightlies and that
means installing manually without an ebuild.  I've always avoided this
because I don't want files spread across my system without an easy way
to remove them.  I've also always wanted to set up a good cruft
removal script for keeping my system clean (I'll admit that takes me
in the hobbyist direction) and I'm thinking the two might work well
together.  I could temporarily install apps without an ebuild and use
a cruft removal script to remove them.

What do you guys think of this?  Do you know of a good cruft removal script?

- Grant

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

On Mon, 26 Jan 2009 08:41:52 -0800
Grant <emailgrant@gmail.com> wrote:

> What do you guys think of this?  Do you know of a good cruft removal script?

Yep, there's quite good one in gentoo itself.

Basically, you'll need to write a short config for it, consisting of
lines like "cruft name", "cruft src uri" and a few more lines if you'll
need to pass some extra parameters to configure/make/install.
It'll build the package in a sandbox, then transfer it to destination,
memorizing every change it did and preventing collisions and config
overwrites.

Just put that config script into an ebuild file and use portage to
build it - as simple as it gets ;)

-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Jesús Guerrero]

On Mon, January 26, 2009 17:48, Mike Kazantsev wrote:
> On Mon, 26 Jan 2009 08:41:52 -0800
> Grant <emailgrant@gmail.com> wrote:
>
>> What do you guys think of this?  Do you know of a good cruft removal
>> script?
>
> Yep, there's quite good one in gentoo itself.
>
> Basically, you'll need to write a short config for it, consisting of
> lines like "cruft name", "cruft src uri" and a few more lines if you'll
> need to pass some extra parameters to configure/make/install.
> It'll build the package in a sandbox, then transfer it to destination,
> memorizing every change it did and preventing collisions and config
> overwrites.
>
> Just put that config script into an ebuild file and use portage to
> build it - as simple as it gets ;)

If you have to learn to write anything, I suggest learning to write ebuilds
instead which will probably be a better long term solution. That way you
can reuse that knowledge in your Gentoo experience and you don't need
a cruft remover either because you can just emerge -C your program.

-- 
Jesús Guerrero

Re: [gentoo-user] Installing outside of Portage & cruft removal

[b.n.]

Jesús Guerrero ha scritto:
> On Mon, January 26, 2009 17:48, Mike Kazantsev wrote:
>> On Mon, 26 Jan 2009 08:41:52 -0800
>> Grant <emailgrant@gmail.com> wrote:
>>
>>> What do you guys think of this?  Do you know of a good cruft removal
>>> script?
>> Yep, there's quite good one in gentoo itself.
>>
>> Basically, you'll need to write a short config for it, consisting of
>> lines like "cruft name", "cruft src uri" and a few more lines if you'll
>> need to pass some extra parameters to configure/make/install.
>> It'll build the package in a sandbox, then transfer it to destination,
>> memorizing every change it did and preventing collisions and config
>> overwrites.
>>
>> Just put that config script into an ebuild file and use portage to
>> build it - as simple as it gets ;)
> 
> If you have to learn to write anything, I suggest learning to write ebuilds
> instead which will probably be a better long term solution. That way you
> can reuse that knowledge in your Gentoo experience and you don't need
> a cruft remover either because you can just emerge -C your program.
> 

Ehm, that's what Mike was suggesting, albeit wittily :)

m.

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

>> What do you guys think of this?  Do you know of a good cruft removal script?
>
> Yep, there's quite good one in gentoo itself.
>
> Basically, you'll need to write a short config for it, consisting of
> lines like "cruft name", "cruft src uri" and a few more lines if you'll
> need to pass some extra parameters to configure/make/install.
> It'll build the package in a sandbox, then transfer it to destination,
> memorizing every change it did and preventing collisions and config
> overwrites.
>
> Just put that config script into an ebuild file and use portage to
> build it - as simple as it gets ;)

I suppose you and Jesus are right, but what about cruft removal?  Are
you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
and especially ~/.*

- Grant

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]

On Mon, 26 Jan 2009 09:35:13 -0800
Grant <emailgrant@gmail.com> wrote:

> I suppose you and Jesus are right, but what about cruft removal?  Are
> you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
> and especially ~/.*

Oh, that's not a bug, it's a feature (C) some_unknown_company

AFAIK portage won't remove any changed (by hand) files in
CONFIG_PROTECT paths, and I've certainly can't remember a package,
which installs anything into ~.
Of course, if you'll launch installed app it'll create some config,
temp files, etc, but that'll only affect the paths to which it has
access, and that's probably only your ~ and /tmp.

And as soon as portage will start to "clean up" your home from
app-created files like, say, some documents you've spent weeks working
on, it'll certainly make it to the top-10 evil linux viruses out there.


-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

On Mon, 26 Jan 2009 22:57:13 +0500, Mike Kazantsev wrote:

> AFAIK portage won't remove any changed (by hand) files in
> CONFIG_PROTECT paths,

Portage won't remove any files in CONFIG_PROTECTed paths,changed ornot.
It also won't remove any files that have been changed since installation,
wherever they were installed.


-- 
Neil Bothwick

When puns are outlawed only outlaws will have puns.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Dale]

Grant wrote:
>>> What do you guys think of this?  Do you know of a good cruft removal script?
>>>       
>> Yep, there's quite good one in gentoo itself.
>>
>> Basically, you'll need to write a short config for it, consisting of
>> lines like "cruft name", "cruft src uri" and a few more lines if you'll
>> need to pass some extra parameters to configure/make/install.
>> It'll build the package in a sandbox, then transfer it to destination,
>> memorizing every change it did and preventing collisions and config
>> overwrites.
>>
>> Just put that config script into an ebuild file and use portage to
>> build it - as simple as it gets ;)
>>     
>
> I suppose you and Jesus are right, but what about cruft removal?  Are
> you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
> and especially ~/.*
>
> - Grant
>
>
>   

Portage doesn't remove files from /etc and should never touch any home
directory.  As far as I know, portage never puts anything in a home
directory either.  You have to clean out /etc and home directories yourself.

Portage does do a good job of removing all the other files tho.  If it
puts it there, it will remove it if you unmerge a package.

Dale

:-)  :-)

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

>>>> What do you guys think of this?  Do you know of a good cruft removal script?
>>>>
>>> Yep, there's quite good one in gentoo itself.
>>>
>>> Basically, you'll need to write a short config for it, consisting of
>>> lines like "cruft name", "cruft src uri" and a few more lines if you'll
>>> need to pass some extra parameters to configure/make/install.
>>> It'll build the package in a sandbox, then transfer it to destination,
>>> memorizing every change it did and preventing collisions and config
>>> overwrites.
>>>
>>> Just put that config script into an ebuild file and use portage to
>>> build it - as simple as it gets ;)
>>>
>>
>> I suppose you and Jesus are right, but what about cruft removal?  Are
>> you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
>> and especially ~/.*
>>
>> - Grant
>>
>>
>>
>
[snip]
> You have to clean out /etc and home directories yourself.

Exactly.  Nothing to help me along?  I'll check out qfile, but I'm
surprised there isn't a good script for this.

I'm the only one interested in a filesystem audit?

- Grant


> Portage does do a good job of removing all the other files tho.  If it
> puts it there, it will remove it if you unmerge a package.
>
> Dale

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Dale]

[-- Attachment #1: Type: text/plain, Size: 1741 bytes --]

Grant wrote:
>>>>> What do you guys think of this?  Do you know of a good cruft removal script?
>>>>>
>>>>>           
>>>> Yep, there's quite good one in gentoo itself.
>>>>
>>>> Basically, you'll need to write a short config for it, consisting of
>>>> lines like "cruft name", "cruft src uri" and a few more lines if you'll
>>>> need to pass some extra parameters to configure/make/install.
>>>> It'll build the package in a sandbox, then transfer it to destination,
>>>> memorizing every change it did and preventing collisions and config
>>>> overwrites.
>>>>
>>>> Just put that config script into an ebuild file and use portage to
>>>> build it - as simple as it gets ;)
>>>>
>>>>         
>>> I suppose you and Jesus are right, but what about cruft removal?  Are
>>> you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
>>> and especially ~/.*
>>>
>>> - Grant
>>>
>>>
>>>
>>>       
> [snip]
>   
>> You have to clean out /etc and home directories yourself.
>>     
>
> Exactly.  Nothing to help me along?  I'll check out qfile, but I'm
> surprised there isn't a good script for this.
>
> I'm the only one interested in a filesystem audit?
>
> - Grant
>
>
>   
>> Portage does do a good job of removing all the other files tho.  If it
>> puts it there, it will remove it if you unmerge a package.
>>
>> Dale
>>     
>
>
>   

There is a script on the forums that is supposed to clean out /etc.  I
am attaching a copy of it.  I have no idea if it still works or if it
will completely destroy your system.  I would search for the thread or
go through the script to make sure it doesn't mess up something.

I do wish there was a option for cleaning out /etc but then again, that
could be dangerous too. 

Dale

:-)  :-) 

[-- Attachment #2: etc-portclean --]
[-- Type: text/plain, Size: 25347 bytes --]

 #!/bin/bash

VERSION=3.11

###########################
# etcportclean
# Isaac Chanin, 2005
#
# Refer to 'etcportclean --help' for information.
#
# This code is not copyrighted and is in the public domain.
# It comes with no warranty of any kind, explicit or otherwise.
###########################

# Exit codes
SUCCESS=0
NOT_ROOT=10
BAD_ARGS=20
WORLD_CHECK_FAIL=30
SANITY_CHECK_FAIL=31
DUPLICATE_FAIL=40
DUPLICATE_WARN=41
NO_FILES=50
MISSING_DEPENDENCY=51
CANNOT_DOWNLOAD=60
BAD_HASH=61

# Directory where portage files are kept
FILES_DIR="/etc/portage"

# Where etcportclean is on the web
WEB=users.wpi.edu/~chanin/scripts/etcportclean

# Files we (optionally) look at
files=( "package.keywords" "package.unmask" "package.mask" "package.use" "package.cflags" "package.cxxflags" "package.ldflags" )

# Messages about files
messages=( "unnecessary keywords" "needlessly unmasked packages" "needlessly masked packages" "unnecessary use flags" "repeat cflags" "repeat cxxflags" "repeat ldflags" )

# Generally needed progs
neededprogs=( "sed" "awk" "grep" "egrep" "sort" "uniq" "tail" "head" "xargs" )

if [ "$UID" -ne 0 ] ; then
	echo "You must be root to run this script."
	exit $NOT_ROOT
fi

function usage()
{
	echo "etcportclean v$VERSION - clean, check and stabilize $FILES_DIR/package.*"
	echo
	echo "  usage:"
	echo "  $prog [flags]"
	echo
	echo "  Flags may be any of:"
	echo "   -cNNNNNNN, -rN, -wN, -dN, -v[N], -V[N], -u[N]"
	echo "  Where N is either 0 or 1."
	echo
	echo "  -c Check: This flag allows you to indicate, on the command line, which"
	echo "  files you would like to check.  The order is as follows (package.):"
	echo "  `echo "${files[@]}" | sed 's/\ /\n/g' | awk -F . '{ print $2 " " }' | xargs`."
	echo "  So, for example, -c0011000 would check ${files[2]} and ${files[3]}."
	echo "  Defaults to asking the user (unless the file does not exist.)"
	echo
	echo "  -r Remove: This flag allows you to toggle automatic removal of unnecessary"
	echo "  entries.  It only affects keywords, unmask, and mask checking.  The others do"
	echo "  not have any automatic removal mechanism. Defaults to asking the user, if"
	echo "  there is anything to remove."
	echo
	echo "  -w World Check: This flag allows you to turn 'emerge -puD world' checking on"
	echo "  or off.  It defaults to on if you are checking any of keywords unmask or mask."
	echo "  NOTE: Turning off 'emerge -puD world' checking may cause etcportclean to miss"
	echo "  lines that could otherwise be removed. Defaults to on."
	echo
	echo "  -d Duplicate Check: This flag allows you to turn duplicate checking on or off."
	echo "  It does not affect use, cflags, cxxflags or ldflags duplicate checking."
	echo "  NOTE: For certain exotic configurations you may have to turn off duplicate"
	echo "  checking.  One example would be if you have multiple accepted keywords for"
	echo "  a single package.  Defaults to on."
	echo
	echo "  -v Verbose: This flag makes etcportclean verbose about what it is doing.  If"
	echo "  you find what you think is a bug, try to get some more information by using"
	echo "  this or -V.  Defaults to off."
	echo
	echo "  -V Very Verbose: This flag makes etcportclean even more verbose (implies -v)."
	echo "  This was added mainly so I could find bugs.  If you like tons of useless"
	echo "  output you may find it useful also. Defaults to off."
	echo
	echo "  -u Update: This flag has etcportclean check for a new version and"
	echo "  automatically update if a new version is found.  It requires sha1sum and wget"
	echo "  with https support (to get the hash file.)  Defaults to off."
	echo
	echo "  etcportclean also allows you to specify a section in keywords, unmask or mask"
	echo "  where it will not check.  This can be useful if you want to always use the"
	echo "  testing version of a package.  In order to use this feature simply place a"
	echo "  comment with the word 'Keep' somewhere in the file.  etcportclean will then"
	echo "  not look above that point in the file for normal checks.  The lines above the"
	echo "  'Keep line' may still trigger duplicate or other warnings, however."
	echo
	exit $BAD_ARGS
}

# Prompt a yes or no question
function readyesno
{
	local message=$1
	unset yesno
	while [ -z "`echo "$yesno" | egrep -i "^(yes|y|no|n)$"`" ]
	do
		echo "$1 (yes/no)"
		read yesno
	done
	if [ -n "`echo "$yesno" | egrep -i "^(yes|y)$"`" ] ; then
		return 1
	else
		return 0
	fi
}

# Checks that an emerge didn't get completely incorrect results because of a masked package.
function checksanity
{
	local emergelist="$1"
	local msg="$2"
	local file=$3
	if [[ "$VERBOSE" ]] ; then echo "Checking for emerge sanity over $file..." ; fi
	# If we get a 'masked notice' instead of a ebuild list
	if [ -n "`echo "$emergelist" | grep "^.*\ satisfy\ .*\.$"`" ]
	then
		mv -f $file.bak.keep $file 2>/dev/null
		echo "Sanity check failed for $file, there appears to be a masking/existence problem."
		echo "$msg"
		exit $SANITY_CHECK_FAIL
	fi
	if [[ "$VERBOSE" ]] ; then echo "Emerge sanity confirmed." ; fi
}

# Make sure we get the output we're going to be looking for from 'emerge -puD world'
function checkworld
{
	if [[ "$VERBOSE" ]] ; then echo "Checking emerge -puD world for expected results..." ; fi
	if [ -n "`emerge -puD world | egrep "\[ebuild\ [A-Z\ ]{6}\]"`" ]
	then
		echo "'emerge -puD world' gives output containing ebuilds on your machine."
		echo "etcportclean relies upon 'emerge -puD world' not changing any packages. It is"
		echo "recommended that you 'emerge -uD world' and take any other needed actions so"
		echo "'emerge -puD world' does not list any ebuilds before using this script. If"
		echo "you do not, scans will not produce incorrect results, but may not be able to"
		echo "find everything that is removable."
		if readyesno "Would you like to continue?"
		then
			exit $WORLD_CHECK_FAIL
		fi
	elif [[ "$VERBOSE" ]] ; then
		echo "emerge -puD world produced expected results."
	fi
}

# Check for duplicate package lines in a package.* file
function checkrdups
{
	local file=$1

	if [[ "$VERBOSE" ]] ; then echo "Checking for duplicate package lines in $file..." ; fi

	declare -a seenpkgs
	local dindex=0
	local seen=0

	# This could probably use getpkgs
	for i in `cat $file | awk '{ print $1 }' | sed 's/[~><=]//g' | sed 's/^\(.*\/.*\)-[0-9]\{1,\}.*$/\1/g' | grep ^[A-Za-z][^\ ]*[A-Za-z]$ | sort | xargs`
	do
		seen=0
		if [[ "$VV" ]] ; then echo "Looking at $i..." ; fi
		for j in ${seenpkgs[@]}
		do
			# Check for duplicate package lines
			if [[ "$j" == "$i" ]] ; then
				seen=1
				echo "Duplicate line for $i in $file.  May want to do something about it."
				break
			fi
		done
		if [[ "$seen" == "0" ]] ; then
			seenpkgs[$dindex]=$i
			let "dindex++"
			# Check if category exists if this is a category line
			if [ -n "`echo $file | grep flag`" -a -z "`echo $i | grep \/`" ] ; then
				if ! [ -d "/usr/portage/$i" ] ; then
					echo "$i is in $file but does not seem to exist in portage."
				fi
			fi
		fi
	done
}

# Check for duplicate modifiers in a package. file (not for package.use)
function checkdups
{
	local file=$1

	if [[ "$VERBOSE" ]] ; then echo "Checking for duplicates in $file..." ; fi
	
	# Array of modifiers for one package
	declare -a npkgflgs
	
	# Look through each package's modifiers
	getflags $file
	for i in "${flags[@]}"
	do
		i="`echo $i | sed 's/@/\ /g'`"
		if [[ "$VV" ]] ; then echo "Looking at $i..." ; fi
		# If there is more than one modifier
		if [ -n "`echo "$i" | awk '{ print $3 }'`" ]
		then
			# Extract package
			local pkg="`echo "$i" | awk '{ print $1 }' | awk -F / '{ print $1 "\\\\/" $2 }'`"
			# Extract package's modifiers
			local npkgflgs=( `echo $i | sed -e 's/\ /\n/g' -e '/^$/d' -e "s/$pkg\n//g" | sort | uniq | xargs` )
			# Find where the package's lines are in $file
			local olines="`egrep -n "$pkg(\ |\-[0-9])" $file | grep -v '^[\ ]*#.*' | sed 's/\(.*\)#.*/\1/g' | awk -F : '{ print $1 }' | xargs`"
			if [ -n "`echo "$olines" | grep '[0-9]'`" ] ; then
				echo "$file has duplicate lines for `echo "$i" | awk '{ print $1 }'`."
				echo "The offending lines are $olines."
				echo "The conflicting flags are ${npkgflgs[@]}."
				echo "You may run the script with the -d0 flag if you know what you're doing."
				echo "(Read --help for a bit more information.)"
				if [[ "${#npkgflgs[@]}" > 1 ]] ; then
					echo "You MUST fix this in order to use this script!"
					exit $DUPLICATE_FAIL
				else
					echo "You SHOULD fix this, though it is valid and won't affect this script."
					if readyesno "Would you like to continue?"
					then
						exit $DUPLICATE_WARN
					fi
				fi
			fi
		fi
	done
	if [[ "$VV" ]] ; then echo ; fi
}

# Gets package modifiers from a file.  Moves all modifiers to one package line.
function getflags
{
	local file=$1
	if [[ "$VERBOSE" ]] ; then echo "Getting package modifiers from $file..." ; fi
	flags=( `egrep "^([a-z]|(<|>|=|~))+.+/[A-z]+.+$" $file | sed -e 's/^\(.*\/[_A-z0-9\-]*\)[^\ ]*\(.*\)/\1\2/' -e 's/\-[0-9]\ /\ /g' | awk '{i=$1;$1="";a[i]=a[i]$0}END{for(i in a)print i a[i]}' | sort | sed -e 's/\(.*\)#.*/\1/g' -e 's/\ /\@/g' | xargs` )
}

# Gets category modifers from a file.  Moves all modifiers to one category line.
function getflags4cats
{
	local file=$1
	if [[ "$VERBOSE" ]] ; then echo "Getting category modifers from $file..." ; fi
	flags=( `egrep "^[a-z]+[^/]*$" $file | awk '{i=$1;$1="";a[i]=a[i]$0}END{for(i in a)print i a[i]}' | sort | sed -e 's/\(.*\)#.*/\1/g' -e 's/\ /\@/g' | xargs` )
}

# Gets packages from a package.* file.
function getpkgs
{
	local file=$1
	if [[ "$VERBOSE" ]] ; then echo "Getting packages from $file..." ; fi
	# Work around if there is no 'Keep' section in the file
	if [ -n "`grep Keep $file`" ] ; then
		local tv="`cat $file | sed -n '/.*Keep.*/,$p'`"
	else
		local tv="`cat $file`"
	fi
	# Array of packages listed in $file
	packages=( `echo "$tv" | egrep -v "^[\ ]*#.*" | awk '{ print $1 }' | egrep "^([a-z]|(<|>|=|~))+.+/[A-z]+.+$" | sed -e 's/^[=|<|>]*\(.*\)$/\1/g' -e 's/^\(.*\)-[0-9]\{1,\}.*$/\1/g' | sort | uniq | xargs` )
}

# Gets categories from a package.* file.
function getcats
{
	local file=$1
	if [[ "$VERBOSE" ]] ; then echo "Getting categories from $file..." ; fi
	# Array of categories listed in $file
	categories=( `tac $file | egrep -v "^[\ ]*#.*" | awk '{ print $1 }' | egrep "^[a-z]+[^/]*$" | sort | uniq | xargs` )
}

# Sets s to a flag's antiflag
function antiflag
{
	flag=$1
	if [ -n "`echo "$flag" | grep '^\-[^\ ]+$'`" ] ; then
		s="`echo "$j" | sed 's/\-\([^\ ]*\)/\1/g'`"
	else
		s="`echo "$j" | sed 's/\([^\ ]*\)/\\\-\1/g'`"
	fi
}

# Check the flags of a file for sanity, requires dupport, temparr, and dindex
function checkflags
{
	local file=$1
	local package=$2
	local flag=$3
	local remgroup=$4

	if [[ "$VV" ]] ; then echo "Checking flag $flag..." ; fi

	if [ -n "`echo "$remgroup" | grep 'ebuild'`" ] ; then
		userg=1
	else
		userg=0
	fi

	# Check if flag is needed
	if [[ "$userg" == 1 ]] ; then
		if [[ "$VV" ]] ; then echo "Checking if $flag is already implied..." ; fi
		local jj="`echo "$flag" | sed 's/^[\-]\{0,1\}\(.*\)$/\1\\\*/g'`"
		if [ -n "`echo "$remgroup" | grep "\[ebuild\ " | grep $package | sed 's/^\[ebuild\ \([A-Z\ ]*\)\ \]\ .*/\1/g' | grep [NU]`" ] ; then
			if [[ "$VERBOSE" ]] ; then echo "Cannot determine status of $flag for $package." ; fi
		else
			if [ -z "`echo $remgroup | egrep $jj`" ] ; then echo "For package $package flag $flag may be removed." ; fi
		fi
	fi
	# Get flag's anti-flag
	antiflag "$flag"
	# Check if anti-flag is set as well
	if [ -n "`echo "${temparr[@]}" | egrep "(^$s|\ $s\ |\ $s$)"`" ] ; then
		local olines="`grep -n "$package" $file.bak.keep | awk -F : '{ print $1 }' | xargs`"
		echo "Conflicting flag $flag for $package in $file on line(s) $olines."
	fi
	# Check for duplicate flags
	local escj=`echo "$flag" | sed 's/\-/\\\-/g'`
	local goon=0
	for k in "${dupport[@]}"
	do
		if [[ "$VV" ]] ; then echo "Comp: $k v. $flag" ; fi
		if [[ "$k" == "$flag" ]] ; then local goon=1 ; fi
	done
	if [[ "$VV" ]] ; then echo "Setting Reapeats[$dindex] to $flag." ; fi
	dupport[dindex]="$flag"
	let "dindex++"
	if [ "$goon" -eq "1" ]
	then
		local olines="`grep -n "$package" $file.bak.keep | grep "$escj" | awk -F : '{ print $1 }' | xargs`"
		echo "Duplicate flag $flag for $package in $file on line(s) $olines."
	fi
}

# Checks individual flags for each package in package.use or package.c*flags.
# It does not offer an option to remove them if they are not needed.
function othercheck
{
	unset packages
	local file=$1
	# Stores per-package useflags
	declare -a temparr
	# Stores duplicate reported useflags
	declare -a dupport

	cd $FILES_DIR
	echo
	checkrdups $file
	getpkgs $file
	getcats $file
	getflags $file
	local index=0
	dindex=0
	if [[ "$file" == "package.use" ]] ; then local usef=1 ; fi

	# For determining if package is installed
	if [[ "$VERBOSE" ]] ; then echo "Checking for masking/existence problems over $file..." ; fi
	local emerges="`emerge -p \`echo "${packages[@]}"\``"
	local mpkg="`echo $emerges | grep 'satisfy\ .*\.' | awk -F \\\" '{ print $2 }'`"
	checksanity "$emerges" "Please confirm that $mpkg exists and is not masked." $file
	local emerges="`echo "$emerges" | grep "\[ebuild\ "`"
	mv -f $file $file.bak.keep

	echo "Checking for unnecessary entries in $file..."
	echo

	# For each package
	if [[ "$VERBOSE" ]] ; then echo "Looking at packages..." ; fi
	for i in "${packages[@]}"
	do
		if [[ "$VV" ]] ; then echo "Looking at $i..." ; fi
		# Check if package is installed
		if [ -z "`echo "$emerges" | egrep \"[B\ ]N\ [A-Z\ ]{3}\]\ $i\"`" ]
		then
			if [[ "$VV" ]] ; then echo "Package seems to be installed..." ; fi
			# Get package's emerge -pv string
			if [[ "$usef" == 1 ]] ; then local empvs="`emerge -pv "$i"`" ; fi
			# Extract flags into an array
			temparr=( `echo "${flags[index]}" | sed -e 's/\@/\ /g' -e 's/^[^\ ]* \(.*\)$/\1/g'` )
			# For each flag of that package
			dindex=0
			dupport=()
			for j in "${temparr[@]}"
			do
				if [[ "$j" != "--param" ]] ; then checkflags "$file" "$i" "$j" "$empvs" ; fi
			done
			unset temparr
		else
			echo "$i is in $file but is not installed."
		fi
		let "index++"
		if [[ "$VV" ]] ; then echo ; fi
	done
	local index=0
	# For each category (for package.*flags files)
	getflags4cats $file.bak.keep
	if [[ "$VERBOSE" ]] ; then echo "Looking at categories..." ; fi
	for i in "${categories[@]}"
	do
		if [[ "$VV" ]] ; then echo "Looking at $i..." ; fi
		temparr=( `echo "${flags[index]}" | sed -e 's/\@/\ /g' -e 's/^[^\ ]* \(.*\)$/\1/g'` )
		dindex=0
		dupport=()
		for j in "${temparr[@]}"
		do
			if [[ "$j" != "--param" ]] ; then checkflags "$file" "$i" "$j" ; fi
		done
		let "index++"
		unset temparr
	done
	mv -f $file.bak.keep $file
}

# Finds unnecessary entries and (optionally) removes them (for all but package.use/*flags)
function nonusecheck
{
	unset packages remove
	local file=$1
	local auto=$2
	local checkdup=$3

	echo
	cd $FILES_DIR
	checkrdups $file
	if [[ "$checkdup" != 0 ]] ; then checkdups $file ; fi
	getpkgs $file
	echo "Checking for unnecessary entries in $file..."
	echo
	
	mv -f $file $file.bak.keep
	declare -a remove
	declare -a iversions
	local index=1

	# Find obvious downgrades
	if [[ "$VERBOSE" ]] ; then echo "Checking for obvious cases..." ; echo ; fi
	local obvious="`emerge -puD world | egrep "\[ebuild[\ ]{5}UD\]"`"

	# For each package in the file
	for i in "${packages[@]}"
	do
		cat $file.bak.keep | grep -v "$i" > $file
		i="`echo "$i" | sed 's/^[^a-z]*\([a-z].*\)$/\1/g'`"
		if [[ "$VV" ]] ; then echo "Looking at $i..." ; fi
		# If it's in obvious, its line is necessary
		if [ -z "`echo "$obvious" | grep $i`" ]
		then
			if [[ "$VV" ]] ; then echo "Passed obvious check..." ; fi
			# What we get when we try to emerge it without its line
			local raw=`emerge -p "$i"`
			local es=`echo "$raw" | grep "\[ebuild\ " | grep "$i"`
			if [ -n "$es" ]
			then
				if [[ "$VV" ]] ; then echo "Passed verioning check..." ; fi
				local continu=1
				# Make sure package is a candidate for slotting before bothering with check
				if [ -n "`echo "$es" | sed 's/^\[ebuild\ \([^]]*\)].*/\1/g' | grep 'S'`" ]
				then
					if [[ "$VV" ]] ; then echo "Package looks to be a slot candidate..." ; fi
					# Vars for working with slotted packages
					local category="`echo "$es" | sed 's/^.*\ \(.*\)\/.*$/\1/g'`"
					local version="`echo "$es" | sed 's/^.*-\([0-9]\{1,\}[^\ ]*\)\(.*$\|$\)/\1/g'`"
					local package="`echo "$es" | sed 's/^.*\/\(.*\)-[0-9]\{1,\}.*$/\1/g'`"
					local iversions=( `ls /var/db/pkg/$category/ | sed 's/\ /\n/g' | grep "$package" | sed 's/^.*-\([0-9]\{1,\}.*\)$/\1/g' | xargs` )
					if [[ "$VV" ]] ; then echo "Cat: $category, Ver: $version, Pak: $package, Ivers: ${iversions[@]}" ; fi
					# Make sure that we are not just emerging an already installed slotted version
					local where=0
					local this=0
					for j in "${iversions[@]}"
					do
						let "where++"
						if [[ "$j" == "$version" ]] ; then this=$where ; fi
					done
					if [ $where -gt $this ] ; then
						local continu=0
						if [[ "$VV" ]] ; then echo "Failed slot check ($where > $this)." ; fi
					else
						if [[ "$VV" ]] ; then echo "Passed slot check ($where <= $this)..." ; fi
					fi
				elif [[ "$VV" ]]
				then
					echo "Passed slot check (implcity)..."
				fi
				# If we would get the same version without the keyword/unmasking or if the package
				# is not installed.  We cannot group these emerges because of masking problems.
				if [ -n "`echo "$es" | egrep \"(\ [B\ ]\ R[F\ ]{3}]|\ [B\ ]N[\ ]{4}\]|o\ satisfy)\"`" ]
				then
					remove[index]=$i
					let "index++"
					if [ "$continu" -eq "1" -a -n "`echo "$es" | egrep \"(\ [B\ ]\ R[F\ ]{3}]|o\ satisfy)\"`" ] ; then
						echo "$i in $file is unnecessary."
					elif [ -z "`echo "$es" | egrep \"(\ [B\ ]\ R[F\ ]{3}]|o\ satisfy)\"`" ] ; then
						echo "$i is in $file but is not installed."
					fi
				elif [ "$continu" -eq "1" ] ; then
					if [[ "$VV" ]] ; then echo "Failed change check." ; fi
				fi
				
			elif [ -n "`echo "$raw" | grep "$i" | grep 'no\ ebuilds\ to\ satisfy\ .*\.'`" ] ; then
				remove[index]=$i
				let "index++"
				echo "$i in $file does not seem to exist in portage."
			else
				if [[ "$VV" ]] ; then echo "Failed versioning check." ; fi
				# Check masked packages to see if they're actually installed or not
				local es="`echo "$raw" | grep '\ could\ satisfy\ ' | grep "$i" | sed 's/^.*\"\(.*\)\".*$/\1/g' | head -n 1 | sed 's/^.*\ \(.*\/.*\)-[0-9]\{1,\}.*$/\1/g'`"
				if [ -n "$es" ]
				then
					local installed="`emerge -p unmerge "$es" | grep "Couldn't find $es"`"
				fi
				if [ -n "$installed" ] ; then
					echo "$i is in $file but is not installed."
					remove[index]=$i
					let "index++"
				fi
				unset installed
			fi
		elif [[ "$VV" ]] ; then
			echo "Failed obvious check."
		fi
		if [[ "$VV" ]] ; then echo ; fi
	done
	doremfunc $file $auto
}

# Prompts (or automatically) removes unnecessary entries from a file.  It looks
# in $remove to find unnecessary packages.  Best not used on package.use/*flags.
function doremfunc
{
	local file=$1
	local auto=$2
	
	# If there was anything to remove ask about removing it
	if [ "${#remove[@]}" -gt 0 ]
	then
		# Handle automation
		if [ -z "$auto" ] ; then
			echo
			readyesno "Remove the unnecessary entries?"
			local yesno=$?
		else
			yesno=$auto
		fi
		
		# If they want to remove entries from this file
		if [ "$yesno" -eq 1 ]
		then
			if [[ "$VERBOSE" ]] ; then echo "Removing entries from $file..." ; fi
			# Move file, and get some backups in case sed chokes
			mv -f $file.bak.keep $file
			cp -f $file $file.bak
			
			# Remove each package's line from file
			result="`cat $file`"
			for i in "${remove[@]}"
			do
				if [[ "$VV" ]] ; then echo "Removing $i..." ; fi
				# Get around sed not liking /'s
				i=`echo "$i" | awk -F / '{ print $1 "\\\\/" $2 }'`
				result="`echo "$result" | sed "/$i/d"`"
			done
			echo "$result" > $file
			
			# Get lines in orginal file
			local lines="`wc \"$file.bak\" | awk '{ print $1 }'`"
			# If file is 0 bytes and orginal was longer than 1 lines use backup
			if [ `ls -l $file | awk '{ print $5 }'` -eq 0 -a "$lines" -gt 1 ] ; then
				echo "There was an error while removing unnecessary entries."
				echo "The backup file for $file has been restored."
				mv -f $file.bak $file
			else
				echo "Unnecessary entries successfully removed."
				rm -f $file.bak
			fi
		else
			# If they don't want to auto-remove
			mv -f $file.bak.keep $file
		fi
	fi
	
	# If there was nothing to remove
	if [ -f "$file.bak.keep" ] ; then
		mv -f $file.bak.keep $file
	fi
}

# Check that neededprogs exist
function checkprogs
{
	for i in "${neededprogs[@]}"
	do
		if [[ "$VV" ]] ; then echo "Checking for usability of $i" ; fi
		if [ -n "`which $i | grep "\ no\ $i\ in"`" ] ; then
			echo "$i is required to use etcportclean.  Please install it or add it to your path."
			exit $MISSING_DEPENDENCY
		fi
	done
}

# Check for an updated version of etcportclean
function update
{
	neededprogs=( "wget" "sha1sum" )
	checkprogs
	quiet="-q"

	if [[ "$VERBOSE" ]] ; then
		quiet=""
		echo "Downloading most recent version of etcportclean..."
	fi
	wget $quiet -t 2 -T 10 -O /tmp/etcportclean.tmp http://$WEB
	wget $quiet -t 2 -T 10 -O /tmp/etcportclean.sha1 http://$WEB.sha1
	if [ -s /tmp/etcportclean.tmp -a -s /tmp/etcportclean.sha1 ] ; then
		if [[ "$VV" ]] ; then echo "Both sha1 and new version downloaded successfully." ; fi
	else
		echo "Could not download the most recent etcportclean.  You could try getting it"
		echo "manually at $WEB, or try -u again later."
		echo
		rm -f /tmp/etcportclean.*
		exit $CANNOT_DOWNLOAD
	fi
	if [[ "$VERBOSE" ]] ; then echo "Checking hashes..." ; fi
	local localhash="`sha1sum /tmp/etcportclean.tmp | awk '{ print $1 }'`"
	local webhash="`cat /tmp/etcportclean.sha1 | grep etcportclean | awk '{ print $1 }'`"
	if [[ "$localhash" != "$webhash" ]]
	then
		echo "WARNING: Incorrect hash found!  The new version of etcportclean"
		echo "may have been tampered with.  Cannot auto-update."
		echo
		exit $BAD_HASH
	fi
	if [[ "$VERBOSE" ]] ; then echo "Checking versions..." ; fi
	chmod +x /tmp/etcportclean.tmp
	local dlversion="`/tmp/etcportclean.tmp --help | head -n 1 | sed 's/^.*\([0-9]\{1\}\.[0-9]\{2\}\)\ .*$/\1/g'`"
	local thisversion="`$0 --help | head -n 1 | sed 's/^.*\([0-9]\{1\}\.[0-9]\{2\}\)\ .*$/\1/g'`"
	if [[ "$dlversion" > "$thisversion" ]]
	then
		echo "Updating etcportclean ($thisversion to $dlversion)..."
		mv -f /tmp/etcportclean.tmp $0
		rm -f /tmp/etcportclean.sha1
		echo "Etcportclean successfully updated."
		echo
		exit $SUCCESS
	else
		rm -f /tmp/etcportclean.*
		echo "You already have the most recent ($thisversion) version of etcportclean."
		echo
		exit $SUCCESS
	fi
}

# Array of answers for which files to check
declare -a ynarray

# Parse command-line arguments
prog=$0
checkworldifnull=1
for i in "$@"
do
	iend=`echo "$i" | tail -c-2`
	case `echo "$i" | head -c2` in
		-v) # Verbose
			if [[ "$iend" == "1" || "$iend" == "v" ]] ; then VERBOSE=1 ; fi
		;;
		-V) # Very verbose
			if [[ "$iend" == "1" || "$iend" == "V" ]] ; then
				VV=1
				VERBOSE=1
			fi
		;;
		-d) # Duplicate check (for all but package.use)
			checkdup=$iend
		;;
		-r) # Automatic removal
			autorem=$iend
		;;
		-u) # Update
			if [[ "$iend" == "1" || "$iend" == "u" ]] ; then doupdate=1 ; fi
		;;
		-w) # Emerge -puD world check
			let "checkworldifnull--"
			if [ -z $iend -o `echo "$i" | tail -c-2` -eq 1 ] ; then checkworld ; fi
		;;
		-c) # Look at (keywords,unmask,mask,use,cflags,cxxflags,ldflags)
			declare -a checkarr
			for j in `seq 0 $((${#files[@]}-1))`
			do
				let "k=j+1"
				checkarr[j]=`echo "$i" | rev | head -c-3 | rev | cut -c$k`
			done
		;;
		*) usage ;;
	esac
done

# Make sure all needed programs are found
checkprogs

# Update if requested
if [[ "$doupdate" == 1 ]] ; then update ; fi

# Ask for which files they want to look at
index=0
for i in "${files[@]}"
do
	# Only bother with asking if the file exists
	if [ -f "$FILES_DIR/$i" ]
	then
		# ...and they didn't specify a choice on the command line
		if [ -z "${checkarr[$index]}" ] ; then
			readyesno "Check $i for ${messages[$index]}?"
			result="$?"
		elif [[ "${checkarr[$index]}" == "0" || "${checkarr[$index]}" == "1" ]] ; then
			result="${checkarr[$index]}"
		else
			usage
		fi
		ynarray[index]=$result
	fi
	let "index++" 
done

# Check 'emerge -puD world' if no preference was indicated and they're looking
# at more than just package.use
if [ $checkworldifnull -eq 1 ]
then
	if [ ${ynarray[0]} -eq 1 -o ${ynarray[1]} -eq 1 -o ${ynarray[2]} -eq 1 ] ; then checkworld ; fi
fi

# Look at selected files
any=0
where=0
for i in "${files[@]}"
do
	# If they said yes
	if [[ "${ynarray[$where]}" == 1 ]]
	then
		any=1
		# Package.(use|cflags|cxxflags|ldflags) special case
		if [[ "$i" != "package.keywords" && "$i" != "package.unmask" && "$i" != "package.mask" ]] ; then
			othercheck "$i" "$autorem" "$checkdup"
		else
			nonusecheck "$i" "$autorem" "$checkdup"
		fi
	fi
	let "where++"
done

# If they said 'no' to all the questions, or they don't have any of the files
if [ "$any" -eq 0 ]
then
	echo 
	echo "You must have at least one of:"
	for i in "${files[@]}"
	do
		echo "  $FILES_DIR/$i"
	done
	echo "in order to use this script."
	exit $NO_FILES
fi

echo 
echo "Done."
exit $SUCCESS

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Paul Hartman]

On Mon, Jan 26, 2009 at 12:17 PM, Grant <emailgrant@gmail.com> wrote:
>>>>> What do you guys think of this?  Do you know of a good cruft removal script?
>>>>>
>>>> Yep, there's quite good one in gentoo itself.
>>>>
>>>> Basically, you'll need to write a short config for it, consisting of
>>>> lines like "cruft name", "cruft src uri" and a few more lines if you'll
>>>> need to pass some extra parameters to configure/make/install.
>>>> It'll build the package in a sandbox, then transfer it to destination,
>>>> memorizing every change it did and preventing collisions and config
>>>> overwrites.
>>>>
>>>> Just put that config script into an ebuild file and use portage to
>>>> build it - as simple as it gets ;)
>>>>
>>>
>>> I suppose you and Jesus are right, but what about cruft removal?  Are
>>> you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
>>> and especially ~/.*
>>>
>>> - Grant
>>>
>>>
>>>
>>
> [snip]
>> You have to clean out /etc and home directories yourself.
>
> Exactly.  Nothing to help me along?  I'll check out qfile, but I'm
> surprised there isn't a good script for this.
>
> I'm the only one interested in a filesystem audit?
>
> - Grant
>
>
>> Portage does do a good job of removing all the other files tho.  If it
>> puts it there, it will remove it if you unmerge a package.
>>
>> Dale

I think if you enable auditing in the kernel and emerge
sys-process/audit you can see the access history of any file on your
system. (doesn't help after the fact, but going forward...) At least
then you could see which user/program did things to which files. I
think even if a file is missing that you think should have been there
you can do an audit search and see who deleted it. I've never tried it
myself but that's how I understand it. I could be wrong.

And it's not what you asked for, but for cleaning cruft out of
distfiles i use eclean from app-portage/gentoolkit... works great.
Whenever I'm running low on space I run it and it zaps a few gigs of
unneeded stuff without deleting the distfiles of active packages.

Paul

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

> I think if you enable auditing in the kernel and emerge
> sys-process/audit you can see the access history of any file on your
> system. (doesn't help after the fact, but going forward...) At least
> then you could see which user/program did things to which files. I
> think even if a file is missing that you think should have been there
> you can do an audit search and see who deleted it. I've never tried it
> myself but that's how I understand it. I could be wrong.

That could be really handy if you could find out when was the last
time a file was accessed.  If you weren't sure that a file is cruft or
not, you could examine that last access date.

> And it's not what you asked for, but for cleaning cruft out of
> distfiles i use eclean from app-portage/gentoolkit... works great.
> Whenever I'm running low on space I run it and it zaps a few gigs of
> unneeded stuff without deleting the distfiles of active packages.

Yeah, eclean is great.

- Grant

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Rumen Yotov]

On (26/01/09 09:35) Grant wrote:
> >> What do you guys think of this?  Do you know of a good cruft removal script?
> >
> > Yep, there's quite good one in gentoo itself.
> >
> > Basically, you'll need to write a short config for it, consisting of
> > lines like "cruft name", "cruft src uri" and a few more lines if you'll
> > need to pass some extra parameters to configure/make/install.
> > It'll build the package in a sandbox, then transfer it to destination,
> > memorizing every change it did and preventing collisions and config
> > overwrites.
> >
> > Just put that config script into an ebuild file and use portage to
> > build it - as simple as it gets ;)
> 
> I suppose you and Jesus are right, but what about cruft removal?  Are
> you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
> and especially ~/.*
> 
> - Grant
> 
Hi Grant,

Check 'man qfile' - there're examples for finding cruft files.
HTH. Rumen

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Daniel Pielmeier]

[-- Attachment #1.1: Type: text/plain, Size: 1949 bytes --]

Grant schrieb am 26.01.2009 18:35:
> 
> I suppose you and Jesus are right, but what about cruft removal?  Are
> you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
> and especially ~/.*
> 

No it isn't, but this is very hard to automate cruft detection via
scripts as you still have to decide yourself if a file is really needed
or not.

It is very easy to compare all the files on the filesystem with the list
of files controlled by portage. But everything that is left over is not
cruft, because there are files you put somewhere yourself or files that
are not controlled by portage but are essential to the system. So you
have to use some kind of ignorelist to exclude files that are not under
portage control but needed. As you see it is still you who has to decide
if a file is cruft or not.

I have done something minimalistic that evolved over time and that fits
my needs. I long refrained from attaching it here, but as this question
comes up so often I think I will share it. I am not a programming guru
so there might be bugs or other stuff done in a poor way so don't come
here and complain if the script has killed your kitten.

I will attach the cruft script written in perl and as an example the
ignorelist I use for my system and the list of cruft files that remain
as false positives. You have to customize the ignorelist yourself so it
does a proper job on your system. I have tried to make the script user
friendly but it surly isn't so you may have to tinker with it a bit. The
script actually does not delete anything it just presents you a list of
files that contains all files on the filesystem minus the files
controlled by portage and minus the files you have ignored.


There is also another cruft script that you may get from overlays but I
don't remember where. If you search the archives you may find it as it
has already been mentioned here.

Regards,

Daniel

[-- Attachment #1.2: cruft-control.pl --]
[-- Type: text/plain, Size: 10948 bytes --]

#!/usr/bin/perl
#
#
#
###############################################################################
# Script:	cruft-control.pl
# Version:	0.19
# Autor:	Daniel Pielmeier
###############################################################################

use strict;
use diagnostics;
use warnings;
use File::Find;
#use Text::Diff;
use Getopt::Long qw (:config posix_default gnu_compat pass_through
		no_ignore_case_always);


# Directories

my $destdir = ();
my $confdir = ();
my $searchdir = ();
my $vdb_path = ();


# Files

my $cruft = "cruft.txt";
my $oldcruft = "oldcruft.txt";
my $diffcruft = "diffcruft.txt";
my $negcruft = "negcruft.txt";
my $systemfile = "systemfile.txt";
my $contentlist = "contentlist.txt";
my $packagefile = "packagefile.txt";
my $ignorelist = "ignorelist.conf";


# Variables

my $uid = `id -u`; chomp $uid;
my $debug = ();
my $ignoreitem = ();
my @systemfile = my @ignorelist = ();
my @contentlist = ();
my @packagefile = ();
my $prev = ();
my @cruft = ();
my %packagefilecheck = ();
my @negcruft =();
my %systemfilecheck = ();
my $help = "
DESCRIPTION:
This script tries to find files not under the control of the package manager.
The resulting list of files should normally not called cruft as proper
detection of cruft is almost impossible. So this script does not offer
functions to delete any files reported.

In general it searches all files in the filesystem. It uses an
$ignorelist where files and directories can be specified which will be
ignored from the search to avoid obvious false positives.
Then it gets the files which are listed in CONTENTS from vdb_path where the
files installed by the package manager are stored. This two resulting lists
are compared with each other to get the files which are not under the control
of the package manager. The resulting list is stored in the $cruft file.

Additionally if available it compares the results with an older cruft file
called $oldcruft and creates $diffcruft out of it to find out changes
which occuring when merging and unmerging programs. This is useful when you
are not sure about the files listed in an initial search so one can just
track the changes.

Carefully examine $cruft and/or $diffcruft. Deleting wrong files which
are essential for a working system may break it. You are responsible for your
system. This script will just make it easier to to detect files which are
probably not needed on the system.

Optionally it writes debug info in form of $systemfile, $contentlist,
$packagefile and $negcruft.

$systemfile lists all files found in the filesystem excluding the files
and directories from $ignorelist.
$contentlist lists all paths to the CONTENT files only useful to see which
packages are installed.
$packagefile lists all files under the control of the package manager.
$negcruft this are the files which are in $packagefile but not in
$systemfile because they are not present in the filessystem or ignored by
the ignorefile.

CONFIGURATION:
For $0 you configure the path to configuration files which
for now is only $ignorelist. and the path to generated output ($cruft,
$oldcruft, $diffcruft) and debug files ($systemfile, $packagefile,
$contentlist, $negcruft).

By default the generated output and debug files go to the current directory
and configuration files are read from /etc/cruft-control
This paths can be overridden by the environment variables CRUFT_DESTDIR for
generated output and debug files and CRUFT_CONFDIR for configuration files
or on the commandline using the relevant switches.

Order of preference = default path < environment variable < command line

DEPENDENCIES:
dev-lang/perl
modules: File::Find, Getopt::Long
sys-apps/portage: portageq for getting the vdb_path
sys-apps/diffutils: diff for comparing old and new results
alternative to sys-apps/diffutils (Text::Diff (Algorithm::Diff)) currently
deactivated

TODO:
- Implement POD::USAGE.
- Define proper paths for all files read and written.
- Decide wether to include searchpath and pm database in the available
  options or not.
- Decide wether to use sys-apps/diffutils or Text::Diff in combination with
  Algorithm::Diff
";


# Configurable paths

if (defined $ENV{"CRUFT_DESTDIR"}) {
	$destdir = $ENV{"CRUFT_DESTDIR"};
} else {
	$destdir = ("./");
}

if (defined $ENV{"CRUFT_CONFDIR"}) {
	$confdir = $ENV{"CRUFT_CONFDIR"};
} else {
	$confdir = ("/etc/cruftcontrol");
}


# Hardcoded paths

# Path to searchdir. Normally this is the root-dir if you want to search the
# whole system.
$searchdir = ("/");
# Path to portage database which holds information of all installed ebuilds.
$vdb_path = qx (portageq vdb_path);
chomp ($vdb_path);


# Checking for permissions and correct usage so the program can access all
# files and directories.

if ($uid ne "0") {
	print "\e[1;31m[ERROR]\tYou need to be root to use $0 for accessing all\n";
	print "[ERROR]\tdirectories and files!\e[m\n\n";
	&usage();
}

GetOptions ('destdir|d=s' => \$destdir, 'confdir|c=s' => \$confdir,
			#'searchdir|s=s' => \$searchdir, 'vdb_path|v=s' => \$vdb_path,
			'debug|b' => \$debug, 'help|h|?' => sub {usage()});

&usage() if ($#ARGV > -1);

foreach ("$destdir", "$confdir") {
	unless (-d $_) {
		print "\e[1;31m[ERROR]\tThe directory $_ does not exist\e[m\n\n";
	&usage();
	}
}


# Creating a list of all files in the filesystem excluding files and
# directories which belong to an ignorelist.

print "[INFO]\tSearching files on the filesystem ...\n";

if (-f "$confdir/$ignorelist" && -s _ && -T _) {
	open (IGNORELIST,"<$confdir/$ignorelist") || die ("$ignorelist: $!");

	print "[INFO]\tExcluding files from $confdir/$ignorelist\n";

	foreach (<IGNORELIST>) {
		unless ( $_ eq "\n" || $_ =~ "#" ) {
			push (@ignorelist, $_);
		}
	}
	chomp (@ignorelist);
	
	find(\&sysignore, $searchdir);
	sub sysignore() {
		my $count = 0;

		foreach $ignoreitem (@ignorelist) {
			if ("$File::Find::name" eq "$ignoreitem") {
				splice(@ignorelist,$count,1);
				($File::Find::prune = 1);
				return;
			}
			$count++;
		}
		push (@systemfile, "$File::Find::name\n");
	}
} else {
	print "\e[1;33m[WARN]\tThe file $ignorelist does not exist in\n[WARN]\t$confdir\n";
	print "[WARN]\tor has no content. This may result in a big and probably\n";
	print "[WARN]\tuseless cruft list. ;-)\e[m\n";

	find(\&sys, $searchdir);
	sub sys() {
		push (@systemfile, "$File::Find::name\n");
	}
}

@systemfile = sort (@systemfile); 


# Searches the package database for the CONTENTS-files of all installed
# packages. This is the base for getting the list of controlled files.

print "[INFO]\tSearching installed packages ...\n";

find(\&content, $vdb_path);
sub content() {
	push (@contentlist, "$File::Find::name\n") if (/CONTENTS/);
}

@contentlist = sort (@contentlist);


# Creating a list of all files owned by all installed packages.

print "[INFO]\tAssigning files to installed packages ...\n";

foreach (@contentlist) { 
	open (CONTENT, "$_") || die("$_: $!");
	foreach (<CONTENT>) {
		if ((/^sym/) || (/^dir/)) {
			push (@packagefile, "$2\n") if (/(\w+\s)(\S+)/);
		}
		if (/^obj/) {
			push (@packagefile, "$2\n") if (/(\w+\s)(.+)(\s\w+\s\d+$)/);
		}
	}
}

@packagefile = sort (@packagefile);
$prev = "not equal to $packagefile[0]";
@packagefile = grep($_ ne $prev && ($prev = $_, 1), @packagefile);


# Compares the list of the files in the system with the files controlled by
# portage to get unmanaged files.

print "[INFO]\tComparing to get unmanaged files ...\n";

open (CRUFT,">$destdir/$cruft") || die ("$cruft: $!");

@packagefilecheck{@packagefile} = ();

foreach $_ (@systemfile) {
	push (@cruft, $_) unless exists $packagefilecheck{$_};
}

@cruft = sort (@cruft);
print CRUFT @cruft;

print "[INFO]\tList containing cruft written to $destdir/$cruft\n";


# Compares the list of found cruft files with a list of files found in an
# older search.

if (-f "$destdir/$oldcruft" && -s _ && -T _) {
	print "[INFO]\tComparing cruft with old results ...\n";

	#open (OLDCRUFT,"<$destdir/$oldcruft") || die ("$oldcruft: $!");
	#my @oldcruft = <OLDCRUFT>;
	#print "[INFO]\tUsing previous results from $destdir/$oldcruft\n";
	#open (DIFFCRUFT,">$destdir/$diffcruft") || die ("$diffcruft: $!");
	#diff \@oldcruft, \@cruft, {OUTPUT => \*DIFFCRUFT};
	system "diff -u $destdir/oldcruft.txt $destdir/cruft.txt >$destdir/diffcruft.txt";
	
	print "[INFO]\tDifferences to previous run written to $destdir/$diffcruft\n";
} else {
	print "[INFO]\tThe file $oldcruft does not exist in\n";
	print "[INFO]\t$destdir\n[INFO]\tor has no contents.\n";
	print "[INFO]\tNote: For later comparisons you can move $cruft to $oldcruft\n";
	print "[INFO]\tto catch the differences in the file $diffcruft.\n";
	print "[INFO]\tSkipping comparison with old results ...\n\n";
}


# Write debug files.

&debug() if defined $debug;


# Subroutines

# Print usage message.

sub usage() {
	print "USAGE: $0 [options]\n";
	print "  -d|--destdir=DIR\tdirectory for analysis and debug output\n";
	print "\t\t\t[$destdir]\n";
	print "  -c|--confdir=DIR\tdirectory containing $ignorelist\n";
	print "\t\t\t[$confdir]\n";
	#print "  -s|--searchdir=DIR\tbase directory for searching\n";
	#print "\t\t\t[$searchdir]\n";
	#print "  -v|--vdb_path=DIR\tpath to the portage database\n";
	#print "\t\t\t[$vdb_path]\n";
	print "  -b|--debug\t\tgenerate debug output\n";
	print "  -?|-h|--help\t\tprint this lot out\n";
	print $help;
	exit;
}


# Subroutine to generate debug output.

sub debug() {

	open (SYS,">$destdir/$systemfile") || die ("$systemfile: $!");
	print SYS @systemfile;
	print "\e[1;35m[DEBUG]\tAll files from the system excluding the ignored files written to $destdir/$systemfile\e[m\n";

	open (CONTENT,">$destdir/$contentlist") || die ("$contentlist: $!");
	print CONTENT @contentlist;
	print "\e[1;35m[DEBUG]\tPath to files containing package contents written to $destdir/$contentlist\e[m\n";

	open (PACKAGE,">$destdir/$packagefile") || die ("$packagefile: $!");
	print PACKAGE @packagefile;
	print "\e[1;35m[DEBUG]\tAll files controlled by the package manager written to $destdir/$packagefile\e[m\n";

# For reference list the files which are controlled by portage but not in the
# system maybe because they are deleted or ignored by the ignorefile.

	print "\e[1;35m[DEBUG]\tComparing to get negcruft files ...\n";

	@systemfilecheck{@systemfile} = ();

	foreach $_ (@packagefile) {
		push (@negcruft, $_) unless exists $systemfilecheck{$_};
	}

	@negcruft = sort (@negcruft);

	open (NEGCRUFT,">$destdir/$negcruft") || die ("$negcruft: $!");
	print NEGCRUFT @negcruft;
	print "\e[1;35m[DEBUG]\tNegative list of cruft written to $destdir/$negcruft\e[m\n";
}

[-- Attachment #1.3: ignorelist.conf --]
[-- Type: text/plain, Size: 744 bytes --]

# Directories and files which should always be ignored.

/boot
/dev
/home
/lost+found
/media
/mnt
/proc
/root
/sys
/tmp
/usr/src
/lib/modules
/usr/tmp
/var

# Directories and files created by the user or in pkg_post_install plus other
# non cruft.

/etc/portage
/etc/runlevels
/etc/ssl/certs
/lib/rc/console
/lib/rc/init.d
/lib/splash/cache
/usr/i686-pc-linux-gnu/bin
/usr/i686-pc-linux-gnu/lib
/usr/include/GL
/usr/lib/openoffice/share/dict
/usr/share/mime

/usr/lib/perl5/site_perl/5.8.8/i686-linux

/usr/lib/gentoolkit/pym
/usr/lib/gimp/2.0/plug-ins
/usr/lib/portage/pym
/usr/lib/scons-1.0.0/SCons
/usr/share/dia/python
/usr/share/hplip
/usr/share/javatoolkit/pym

/usr/lib/python2.5/site-packages

[-- Attachment #1.4: cruft.txt --]
[-- Type: text/plain, Size: 6213 bytes --]

/
/bin/sh
/etc/.pwd.lock
/etc/adjtime
/etc/asound.state
/etc/blkid.tab
/etc/blkid.tab.old
/etc/conf.d/lm_sensors
/etc/conf.d/net
/etc/csh.env
/etc/cups/ppd/HP_Business_Inkjet_1000.ppd
/etc/cups/printers.conf
/etc/cups/printers.conf.O
/etc/eix-sync.conf
/etc/env.d/02ccache
/etc/env.d/02locale
/etc/env.d/03opengl
/etc/env.d/05binutils
/etc/env.d/05gcc-i686-pc-linux-gnu
/etc/env.d/20java
/etc/env.d/90games
/etc/env.d/99local
/etc/env.d/binutils/config-i686-pc-linux-gnu
/etc/env.d/gcc/.NATIVE
/etc/env.d/gcc/config-i686-pc-linux-gnu
/etc/fstab
/etc/gre.d/gre.conf
/etc/group
/etc/group-
/etc/gshadow
/etc/gshadow-
/etc/gtk-2.0/gdk-pixbuf.loaders
/etc/gtk-2.0/gtk.immodules
/etc/init.d/net.eth0
/etc/java-config-2/current-system-vm
/etc/ld.so.cache
/etc/ld.so.conf
/etc/locale.gen
/etc/localtime
/etc/make.conf
/etc/make.profile
/etc/modprobe.conf
/etc/mtab
/etc/openldap
/etc/openldap/ssl
/etc/openldap/ssl/ldap.crt
/etc/openldap/ssl/ldap.csr
/etc/openldap/ssl/ldap.key
/etc/openldap/ssl/ldap.pem
/etc/pango/pango.modules
/etc/passwd
/etc/passwd-
/etc/printcap
/etc/profile.csh
/etc/profile.env
/etc/resolv.conf
/etc/sgml
/etc/sgml/catalog
/etc/sgml/xml-docbook-4.4.cat
/etc/shadow
/etc/shadow-
/etc/splash/default
/etc/ssh/ssh_host_dsa_key
/etc/ssh/ssh_host_dsa_key.pub
/etc/ssh/ssh_host_key
/etc/ssh/ssh_host_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_rsa_key.pub
/etc/timezone
/etc/udev/rules.d/10-local.rules
/etc/udev/rules.d/70-persistent-cd.rules
/etc/udev/rules.d/70-persistent-net.rules
/etc/xml/catalog
/etc/xml/docbook
/lib/cpp
/lib/libgcc_s.so.1
/lib/rc/cache
/lib/rc/cache/depconfig
/lib/rc/cache/deptree
/lib/rc/cache/nettree
/lib/rc/cache/softlevel
/lib/rcscripts/awk
/lib/rcscripts/awk/fixlafiles.awk
/lib/splash/sys
/opt/Adobe/Help/de_DE/Adobe Reader
/opt/Adobe/Help/de_DE/Adobe Reader/8.0
/opt/Adobe/Help/de_DE/Adobe Reader/8.0/images
/opt/Adobe/Help/en_US/Adobe Reader
/opt/Adobe/Help/en_US/Adobe Reader/8.0
/opt/Adobe/Help/en_US/Adobe Reader/8.0/images
/sbin/fix_libtool_files.sh
/usr/bin/addr2line
/usr/bin/ar
/usr/bin/as
/usr/bin/c++
/usr/bin/c++filt
/usr/bin/c89
/usr/bin/c99
/usr/bin/cc
/usr/bin/cpp
/usr/bin/ctags
/usr/bin/erb
/usr/bin/esd
/usr/bin/ex
/usr/bin/g++
/usr/bin/gcc
/usr/bin/gcov
/usr/bin/gfortran
/usr/bin/gprof
/usr/bin/i686-pc-linux-gnu-addr2line
/usr/bin/i686-pc-linux-gnu-ar
/usr/bin/i686-pc-linux-gnu-as
/usr/bin/i686-pc-linux-gnu-c++
/usr/bin/i686-pc-linux-gnu-c++filt
/usr/bin/i686-pc-linux-gnu-cpp
/usr/bin/i686-pc-linux-gnu-g++
/usr/bin/i686-pc-linux-gnu-gcc
/usr/bin/i686-pc-linux-gnu-gfortran
/usr/bin/i686-pc-linux-gnu-gprof
/usr/bin/i686-pc-linux-gnu-ld
/usr/bin/i686-pc-linux-gnu-nm
/usr/bin/i686-pc-linux-gnu-objcopy
/usr/bin/i686-pc-linux-gnu-objdump
/usr/bin/i686-pc-linux-gnu-ranlib
/usr/bin/i686-pc-linux-gnu-readelf
/usr/bin/i686-pc-linux-gnu-size
/usr/bin/i686-pc-linux-gnu-strings
/usr/bin/i686-pc-linux-gnu-strip
/usr/bin/idle
/usr/bin/irb
/usr/bin/ld
/usr/bin/nm
/usr/bin/objcopy
/usr/bin/objdump
/usr/bin/pydoc
/usr/bin/python
/usr/bin/python-config
/usr/bin/python2
/usr/bin/ranlib
/usr/bin/rdoc
/usr/bin/readelf
/usr/bin/ri
/usr/bin/ruby
/usr/bin/size
/usr/bin/strings
/usr/bin/strip
/usr/bin/testrb
/usr/bin/vi
/usr/bin/view
/usr/bin/yacc
/usr/include/ansidecl.h
/usr/include/bfd.h
/usr/include/bfdlink.h
/usr/include/db.h
/usr/include/db_185.h
/usr/include/dis-asm.h
/usr/include/libiberty.h
/usr/include/qt4/Gentoo/gentoo-qconfig.h
/usr/include/symcat.h
/usr/kde/3.5/share/applications/mimeinfo.cache
/usr/kde/3.5/share/services/ksycoca
/usr/lib/cracklib_dict.hwm
/usr/lib/cracklib_dict.pwd
/usr/lib/cracklib_dict.pwi
/usr/lib/gconv/gconv-modules.cache
/usr/lib/graphviz/config
/usr/lib/libGL.la
/usr/lib/libGL.so
/usr/lib/libGLcore.so
/usr/lib/libXvMCNVIDIA_dynamic.so.1
/usr/lib/libdb.a
/usr/lib/libdb.so
/usr/lib/libdb_cxx.a
/usr/lib/libdb_cxx.so
/usr/lib/libdb_java.a
/usr/lib/libdb_java.so
/usr/lib/libperl.so
/usr/lib/libruby.so
/usr/lib/locale/locale-archive
/usr/lib/moo/plugins/lib/insert_date_and_time.pyc
/usr/lib/moo/plugins/lib/insert_date_and_time.pyo
/usr/lib/moo/plugins/lib/medit/__init__.pyc
/usr/lib/moo/plugins/lib/medit/__init__.pyo
/usr/lib/moo/plugins/lib/medit/runpython.pyc
/usr/lib/moo/plugins/lib/medit/runpython.pyo
/usr/lib/moo/plugins/lib/pyconsole.pyc
/usr/lib/moo/plugins/lib/pyconsole.pyo
/usr/lib/moo/plugins/pycmd.pyc
/usr/lib/moo/plugins/pycmd.pyo
/usr/lib/moo/plugins/python.pyc
/usr/lib/moo/plugins/python.pyo
/usr/lib/moo/plugins/terminal.pyc
/usr/lib/moo/plugins/terminal.pyo
/usr/lib/mozilla-firefox/defaults/autoconfig
/usr/lib/nsbrowser/plugins/javaplugin.so
/usr/lib/perl5/5.8.8/i686-linux/Encode/ConfigLocal.pm
/usr/lib/perl5/vendor_perl/5.8.8/XML/SAX/ParserDetails.ini
/usr/lib/xorg/modules/extensions/libglx.so
/usr/lib/xorg/modules/extensions/libwfb.so
/usr/qt/3/etc/settings/.qt_plugins_3.3rc.lock
/usr/qt/3/etc/settings/qt_plugins_3.3rc
/usr/share/X11/xkb/compiled
/usr/share/applications/mimeinfo.cache
/usr/share/binutils-data/i686-pc-linux-gnu/2.18/info/dir
/usr/share/config
/usr/share/dia/python-startup.pyc
/usr/share/dia/python-startup.pyo
/usr/share/fonts/100dpi/encodings.dir
/usr/share/fonts/100dpi/fonts.dir
/usr/share/fonts/75dpi/encodings.dir
/usr/share/fonts/75dpi/fonts.dir
/usr/share/fonts/Type1/encodings.dir
/usr/share/fonts/Type1/fonts.dir
/usr/share/fonts/Type1/fonts.scale
/usr/share/fonts/misc/encodings.dir
/usr/share/fonts/misc/fonts.dir
/usr/share/gcc-data/i686-pc-linux-gnu/4.1.2/info/dir
/usr/share/icons/Rodent/icon-theme.cache
/usr/share/icons/Tango/icon-theme.cache
/usr/share/icons/hicolor/icon-theme.cache
/usr/share/info/dir
/usr/share/man/man1/ctags.1.bz2
/usr/share/man/man1/ex.1.bz2
/usr/share/man/man1/python.1.bz2
/usr/share/man/man1/vi.1.bz2
/usr/share/man/man1/view.1.bz2
/usr/share/man/whatis
/usr/share/qt4/mkspecs/qconfig.pri
/usr/share/scite/locale.de.properties
/usr/share/services/ksycoca
/usr/share/timidity/current
/usr/share/vim/vim72/doc/gentoo-syntax.txt
/usr/share/xfce4/panel-plugins/mimeinfo.cache

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

>> I suppose you and Jesus are right, but what about cruft removal?  Are
>> you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
>> and especially ~/.*
>>
>
> No it isn't, but this is very hard to automate cruft detection via
> scripts as you still have to decide yourself if a file is really needed
> or not.
>
> It is very easy to compare all the files on the filesystem with the list
> of files controlled by portage. But everything that is left over is not
> cruft, because there are files you put somewhere yourself or files that
> are not controlled by portage but are essential to the system. So you
> have to use some kind of ignorelist to exclude files that are not under
> portage control but needed. As you see it is still you who has to decide
> if a file is cruft or not.
>
> I have done something minimalistic that evolved over time and that fits
> my needs. I long refrained from attaching it here, but as this question
> comes up so often I think I will share it. I am not a programming guru
> so there might be bugs or other stuff done in a poor way so don't come
> here and complain if the script has killed your kitten.
>
> I will attach the cruft script written in perl and as an example the
> ignorelist I use for my system and the list of cruft files that remain
> as false positives. You have to customize the ignorelist yourself so it
> does a proper job on your system. I have tried to make the script user
> friendly but it surly isn't so you may have to tinker with it a bit. The
> script actually does not delete anything it just presents you a list of
> files that contains all files on the filesystem minus the files
> controlled by portage and minus the files you have ignored.
>
>
> There is also another cruft script that you may get from overlays but I
> don't remember where. If you search the archives you may find it as it
> has already been mentioned here.
>
> Regards,
>
> Daniel

Thank you Daniel and Dale and everyone for their opinions regarding
this.  Adding a list of files tracked by portage to a list of files
specified by me and reporting on the different sounds like a very
useful thing to me.  Daniel, can you remember anything else about that
cruft script from an overlay?  I can't seem to come up with it from
Google.

Writing an ebuild is best, but sometimes you just want to give a
program a try without writing an ebuild (like everyone else running
Linux does) and a scruft script enables you to do that without making
a mess of your system.

- Grant

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Jesús Guerrero]

On Mon, January 26, 2009 21:17, Grant wrote:
> Writing an ebuild is best, but sometimes you just want to give a
> program a try without writing an ebuild (like everyone else running
> Linux does) and a scruft script enables you to do that without making
> a mess of your system.

Not to be picky, it's just an idea but in that case, isn't it way easier to
just ./configure --prefix=/some/dir/inside/yourhome or edit a makefile?

-- 
Jesús Guerrero

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

>> Writing an ebuild is best, but sometimes you just want to give a
>> program a try without writing an ebuild (like everyone else running
>> Linux does) and a scruft script enables you to do that without making
>> a mess of your system.
>
> Not to be picky, it's just an idea but in that case, isn't it way easier to
> just ./configure --prefix=/some/dir/inside/yourhome or edit a makefile?

I have to say I know nothing about compiling or installing outside of
portage.  Does specifying a prefix like that work?  You get a fully
functional program with nothing installed outside of some/dir?

- Grant

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Paul Hartman]

On Mon, Jan 26, 2009 at 2:30 PM, Grant <emailgrant@gmail.com> wrote:
>>> Writing an ebuild is best, but sometimes you just want to give a
>>> program a try without writing an ebuild (like everyone else running
>>> Linux does) and a scruft script enables you to do that without making
>>> a mess of your system.
>>
>> Not to be picky, it's just an idea but in that case, isn't it way easier to
>> just ./configure --prefix=/some/dir/inside/yourhome or edit a makefile?
>
> I have to say I know nothing about compiling or installing outside of
> portage.  Does specifying a prefix like that work?  You get a fully
> functional program with nothing installed outside of some/dir?

Yes, it's the "normal way" for people that don't use package managers.
I almost always install into my home directory for programs that
aren't in portage (or make my own ebuild if it is a simple one). Or
depending on what program it is, create a user for it and run it under
that user account so it can't touch anything else.

Also, a lot of more simple programs don't even need to be installed.
Just untar it, configure it, make it and run it from the directory in
which the source resides.

Paul

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Paul Hartman]

On Mon, Jan 26, 2009 at 2:44 PM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Mon, Jan 26, 2009 at 2:30 PM, Grant <emailgrant@gmail.com> wrote:
>>>> Writing an ebuild is best, but sometimes you just want to give a
>>>> program a try without writing an ebuild (like everyone else running
>>>> Linux does) and a scruft script enables you to do that without making
>>>> a mess of your system.
>>>
>>> Not to be picky, it's just an idea but in that case, isn't it way easier to
>>> just ./configure --prefix=/some/dir/inside/yourhome or edit a makefile?
>>
>> I have to say I know nothing about compiling or installing outside of
>> portage.  Does specifying a prefix like that work?  You get a fully
>> functional program with nothing installed outside of some/dir?
>
> Yes, it's the "normal way" for people that don't use package managers.
> I almost always install into my home directory for programs that
> aren't in portage (or make my own ebuild if it is a simple one). Or
> depending on what program it is, create a user for it and run it under
> that user account so it can't touch anything else.
>
> Also, a lot of more simple programs don't even need to be installed.
> Just untar it, configure it, make it and run it from the directory in
> which the source resides.
>
> Paul
>

I should also say that installing that stuff to /usr/local tree is also normal.

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Jesús Guerrero]

On Mon, January 26, 2009 21:44, Paul Hartman wrote:
> On Mon, Jan 26, 2009 at 2:30 PM, Grant <emailgrant@gmail.com> wrote:
>>>> Writing an ebuild is best, but sometimes you just want to give a
>>>> program a try without writing an ebuild (like everyone else running
>>>> Linux does) and a scruft script enables you to do that without making
>>>> a mess of your system.
>>>
>>> Not to be picky, it's just an idea but in that case, isn't it way
>>> easier to
>>> just ./configure --prefix=/some/dir/inside/yourhome or edit a makefile?
>>
>> I have to say I know nothing about compiling or installing outside of
>> portage.  Does specifying a prefix like that work?  You get a fully
>> functional program with nothing installed outside of some/dir?
>
> Yes, it's the "normal way" for people that don't use package managers.
> I almost always install into my home directory for programs that
> aren't in portage (or make my own ebuild if it is a simple one). Or
> depending on what program it is, create a user for it and run it under
> that user account so it can't touch anything else.

Yes. It's mostly that simple. Though sometimes in more complex programs
you might need some extra setup (i.e. configure some variable so the program
can find its path to the required libs or so). It really depends on the
program,
however if you open the readme or install file and take a look 99% of the
times the procedure should be described there. Not all programs use the
tipical make system, so you should always check the docs, and in any case
save the source tree for further refference, or just to be able to make
uninstall.

Note that this is the cleanest method, and in some cases it's the only option
(i.e. you don't have root access to the machine, so you have to build in your
home dir).

>
> Also, a lot of more simple programs don't even need to be installed.
> Just untar it, configure it, make it and run it from the directory in
> which the source resides.

This work for smaller programs very well. Just make and launch it.

-- 
Jesús Guerrero

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

>>>>> Writing an ebuild is best, but sometimes you just want to give a
>>>>> program a try without writing an ebuild (like everyone else running
>>>>> Linux does) and a scruft script enables you to do that without making
>>>>> a mess of your system.
>>>>
>>>> Not to be picky, it's just an idea but in that case, isn't it way
>>>> easier to
>>>> just ./configure --prefix=/some/dir/inside/yourhome or edit a makefile?
>>>
>>> I have to say I know nothing about compiling or installing outside of
>>> portage.  Does specifying a prefix like that work?  You get a fully
>>> functional program with nothing installed outside of some/dir?
>>
>> Yes, it's the "normal way" for people that don't use package managers.
>> I almost always install into my home directory for programs that
>> aren't in portage (or make my own ebuild if it is a simple one). Or
>> depending on what program it is, create a user for it and run it under
>> that user account so it can't touch anything else.
>
> Yes. It's mostly that simple. Though sometimes in more complex programs
> you might need some extra setup (i.e. configure some variable so the program
> can find its path to the required libs or so). It really depends on the
> program,

So for example, miro needs xine to play videos.  If I ./configure miro
with --prefix=/usr/local, it will install to /usr/local/miro or
similar?  Then I would need to point it to xine and possibly others
since it wasn't configured like --prefix=/ ?  Is all this done as
root?

> however if you open the readme or install file and take a look 99% of the
> times the procedure should be described there. Not all programs use the
> tipical make system, so you should always check the docs, and in any case
> save the source tree for further refference, or just to be able to make
> uninstall.

Couldn't I just uninstall with 'rm -rf /usr/local/miro' ?

- Grant


> Note that this is the cleanest method, and in some cases it's the only option
> (i.e. you don't have root access to the machine, so you have to build in your
> home dir).
>
>>
>> Also, a lot of more simple programs don't even need to be installed.
>> Just untar it, configure it, make it and run it from the directory in
>> which the source resides.
>
> This work for smaller programs very well. Just make and launch it.

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Stroller]

On 26 Jan 2009, at 22:51, Grant wrote:
> ...
> So for example, miro needs xine to play videos.  If I ./configure miro
> with --prefix=/usr/local, it will install to /usr/local/miro or
> similar?

Yes. Read the configure options for the app you're installing. It  
might also have a --libprefix or similar that you need to change, too.

> Then I would need to point it to xine and possibly others
> since it wasn't configured like --prefix=/ ?

Usually the configure scripts should find stuff installed in the main  
part of the system.

> Is all this done as root?

`./configure && make` as user, `make install` as root (sudo?).

>> ... in any case
>> save the source tree for further refference, or just to be able to  
>> make
>> uninstall.
>
> Couldn't I just uninstall with 'rm -rf /usr/local/miro' ?

I don't know about miro, but often foo will install not install a  
directory /usr/local/foo but instead files /usr/local/food & /usr/ 
local/foobar. These will get "intermingled" with files /usr/local/bard  
& /usr/local/barfoo, so `make uninstall` is used to uninstall the  
files cleanly.

I believe that configure scripts for some programs (e.g. mplayer?) may  
also sometimes install config files in /etc - I think `make uninstall`  
will remove these, but I get the impression from your earlier posts  
that you may find this undesirable.

Nevertheless, it is worth experimenting with compiling by hand using  
this method - I would consider it an essential Linux skill and it will  
give you an insight into things around which Portage is merely a  
wrapper.

Stroller.

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Hieu, Luu Danh]

If you are installing a package by hand and wants to revert back to
the previous state, best is to :

- when you ./configure it, use the various --prefix directives (do a
./configure --help for information on that)
- when you want to remove, make uninstall in the source dir (so don't
remove it!)
- if it does not have a remove, usually if you install it inside
/home/${username}/whatever, then removing that is fine.

Best thing though is to write an ebuild and then Portage will sandbox
the build so it knows every file that has been installed.

The package knows where to link to when it goes into the ./configure
stage and won't act like windows, installing stuffs into registry or
the like ... everything's nicely contained inside /lib and /share
folders (except /etc files ...which you can safely ignore them there -
those are just text files and you'll know where they are anyway if you
intend to configure miro)

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

> If you are installing a package by hand and wants to revert back to
> the previous state, best is to :
>
> - when you ./configure it, use the various --prefix directives (do a
> ./configure --help for information on that)
> - when you want to remove, make uninstall in the source dir (so don't
> remove it!)
> - if it does not have a remove, usually if you install it inside
> /home/${username}/whatever, then removing that is fine.
>
> Best thing though is to write an ebuild and then Portage will sandbox
> the build so it knows every file that has been installed.
>
> The package knows where to link to when it goes into the ./configure
> stage and won't act like windows, installing stuffs into registry or
> the like ... everything's nicely contained inside /lib and /share
> folders (except /etc files ...which you can safely ignore them there -
> those are just text files and you'll know where they are anyway if you
> intend to configure miro)

Thanks everyone.  I've never been open to manual compile/installation
but I can give it a try now.

- Grant

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Paul Hartman]

On Tue, Jan 27, 2009 at 4:34 PM, Grant <emailgrant@gmail.com> wrote:
>> If you are installing a package by hand and wants to revert back to
>> the previous state, best is to :
>>
>> - when you ./configure it, use the various --prefix directives (do a
>> ./configure --help for information on that)
>> - when you want to remove, make uninstall in the source dir (so don't
>> remove it!)
>> - if it does not have a remove, usually if you install it inside
>> /home/${username}/whatever, then removing that is fine.
>>
>> Best thing though is to write an ebuild and then Portage will sandbox
>> the build so it knows every file that has been installed.
>>
>> The package knows where to link to when it goes into the ./configure
>> stage and won't act like windows, installing stuffs into registry or
>> the like ... everything's nicely contained inside /lib and /share
>> folders (except /etc files ...which you can safely ignore them there -
>> those are just text files and you'll know where they are anyway if you
>> intend to configure miro)
>
> Thanks everyone.  I've never been open to manual compile/installation
> but I can give it a try now.

Once you learn the basics, most programs are the same (configure/make)
and it's not so bad. Obviously the advice to read the README/INSTALL
files is golden, they will almost always tell you what you need to
know.

On my home PC I used to tri-boot OS/2 (my first love), Win95
(wintendo) and Slackware (version 2 or 3?), so back then I think
everything had to be manually configured and compiled pretty much. I
guess it all seems kind of obvious once you already know how to do it.
We've come a long way since then. :)

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Daniel Pielmeier]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

Grant schrieb am 26.01.2009 21:17:
> 
> Thank you Daniel and Dale and everyone for their opinions regarding
> this.  Adding a list of files tracked by portage to a list of files
> specified by me and reporting on the different sounds like a very
> useful thing to me.  Daniel, can you remember anything else about that
> cruft script from an overlay?  I can't seem to come up with it from
> Google.
> 

hollow -> app-admin/findcruft2
gentoo-taiwan -> app-portage/findcruft

Both overlays mentioned above are available via layman but I think
findcruft2 is newer and currently maintained.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

>> Thank you Daniel and Dale and everyone for their opinions regarding
>> this.  Adding a list of files tracked by portage to a list of files
>> specified by me and reporting on the different sounds like a very
>> useful thing to me.  Daniel, can you remember anything else about that
>> cruft script from an overlay?  I can't seem to come up with it from
>> Google.
>>
>
> hollow -> app-admin/findcruft2
> gentoo-taiwan -> app-portage/findcruft
>
> Both overlays mentioned above are available via layman but I think
> findcruft2 is newer and currently maintained.

Thank you, I'll give findcruft2 a try.

- Grant

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Jesús Guerrero]

On Mon, January 26, 2009 18:35, Grant wrote:
>>> What do you guys think of this?  Do you know of a good cruft removal
>>> script?
>>
>> Yep, there's quite good one in gentoo itself.
>>
>> Basically, you'll need to write a short config for it, consisting of
>> lines like "cruft name", "cruft src uri" and a few more lines if you'll
>> need to pass some extra parameters to configure/make/install.
>> It'll build the package in a sandbox, then transfer it to destination,
>> memorizing every change it did and preventing collisions and config
>> overwrites.
>>
>> Just put that config script into an ebuild file and use portage to
>> build it - as simple as it gets ;)
>
> I suppose you and Jesus are right, but what about cruft removal?  Are
> you saying Gentoo is 100% cruft-free?  I've got a lot of junk in /etc
> and especially ~/.*

No. It isn't 100% free.

Emerge will only remove the stuff it installed. There's absolutely no way
to guarantee that any file in /etc is not needed any longer. Besides that:

1.- YOU created those files, so you are the one that should keep the track
of them
2.- They are not part of the package, emerge will only uninstall what it
installed,
nothing else, I would be very angry if a silly package manager starts
deleting files
I created by hand.
3.- you might want to install that package again, and you are probably not
willing
to reconfigure it again
4.- seriously, even three thousand of files in /etc are not a problem.
They take
a few bytes or several kb at most. You better look at /usr/src or the
number of
installed games and/or icon themes.

In the rest of the three there shouldn't be too much cruft if you stick to
portage and don't go installing things by hand around (if you do then you
can't blame portage neither).

Cruft removers base their actions on guesses. I never felt like those
programs
really worked at all. If you use one of them you have still to review the
final list
before removal (that or you are good at making backups, you know). So I don't
know what the point is. I would be quicker to clean the tree myself using
mc and
some bash magic.


-- 
Jesús Guerrero

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Alex Schuster]

Grant wrote:

> I'd like to install the latest miro from their nightlies and that
> means installing manually without an ebuild.  I've always avoided this
> because I don't want files spread across my system without an easy way
> to remove them.  I've also always wanted to set up a good cruft
> removal script for keeping my system clean (I'll admit that takes me
> in the hobbyist direction) and I'm thinking the two might work well
> together.  I could temporarily install apps without an ebuild and use
> a cruft removal script to remove them.
>
> What do you guys think of this?  Do you know of a good cruft removal
> script?

If you can install your apps into a specific location, I'd use xstow.

emerge xstow
cd ~/install/myapp-1.2.3
./configure --prefix=/usr/local/stow/myapp-1.2.3 &&
make &&
make install
cd /usr/local/stow
xstow myapp-1.2.3

myapp-1.2.3 is installed into /usr/local/stow/myapp-1.2.3, but xstow 
created symlinks into the /usr/local hierarchy, so it looks as if it were 
directly installed there. No need to add /usr/local/stow/myapp-1.2.3/bin 
to the PATH and such.
Use xstow -D myapp-1.2.3 to remove the symlinks if you want to uninstall, 
then remove /usr/local/stow/myapp-1.2.3.

	Wonko

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

>> I'd like to install the latest miro from their nightlies and that
>> means installing manually without an ebuild.  I've always avoided this
>> because I don't want files spread across my system without an easy way
>> to remove them.  I've also always wanted to set up a good cruft
>> removal script for keeping my system clean (I'll admit that takes me
>> in the hobbyist direction) and I'm thinking the two might work well
>> together.  I could temporarily install apps without an ebuild and use
>> a cruft removal script to remove them.
>>
>> What do you guys think of this?  Do you know of a good cruft removal
>> script?
>
> If you can install your apps into a specific location, I'd use xstow.
>
> emerge xstow
> cd ~/install/myapp-1.2.3
> ./configure --prefix=/usr/local/stow/myapp-1.2.3 &&
> make &&
> make install
> cd /usr/local/stow
> xstow myapp-1.2.3
>
> myapp-1.2.3 is installed into /usr/local/stow/myapp-1.2.3, but xstow
> created symlinks into the /usr/local hierarchy, so it looks as if it were
> directly installed there. No need to add /usr/local/stow/myapp-1.2.3/bin
> to the PATH and such.
> Use xstow -D myapp-1.2.3 to remove the symlinks if you want to uninstall,
> then remove /usr/local/stow/myapp-1.2.3.

That would still leave anything installed outside of /usr/local (/etc
for example) right?

- Grant

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

Am Montag, 16. Februar 2009 19:33:11 schrieb Grant:

> > If you can install your apps into a specific location, I'd use xstow.
> >
> > emerge xstow
> > cd ~/install/myapp-1.2.3
> > ./configure --prefix=/usr/local/stow/myapp-1.2.3 &&
> > make &&
> > make install
> > cd /usr/local/stow
> > xstow myapp-1.2.3
> >
> > myapp-1.2.3 is installed into /usr/local/stow/myapp-1.2.3, but xstow
> > created symlinks into the /usr/local hierarchy, so it looks as if it were
> > directly installed there. No need to add /usr/local/stow/myapp-1.2.3/bin
> > to the PATH and such.
> > Use xstow -D myapp-1.2.3 to remove the symlinks if you want to uninstall,
> > then remove /usr/local/stow/myapp-1.2.3.
>
> That would still leave anything installed outside of /usr/local (/etc
> for example) right?

No. All the symlinks will be in /usr/local or subdirectories of it (or more 
general: in the parent directory of the stow dir).

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Grant]

>> > If you can install your apps into a specific location, I'd use xstow.
>> >
>> > emerge xstow
>> > cd ~/install/myapp-1.2.3
>> > ./configure --prefix=/usr/local/stow/myapp-1.2.3 &&
>> > make &&
>> > make install
>> > cd /usr/local/stow
>> > xstow myapp-1.2.3
>> >
>> > myapp-1.2.3 is installed into /usr/local/stow/myapp-1.2.3, but xstow
>> > created symlinks into the /usr/local hierarchy, so it looks as if it were
>> > directly installed there. No need to add /usr/local/stow/myapp-1.2.3/bin
>> > to the PATH and such.
>> > Use xstow -D myapp-1.2.3 to remove the symlinks if you want to uninstall,
>> > then remove /usr/local/stow/myapp-1.2.3.
>>
>> That would still leave anything installed outside of /usr/local (/etc
>> for example) right?
>
> No. All the symlinks will be in /usr/local or subdirectories of it (or more
> general: in the parent directory of the stow dir).

I thought running something like '.configure --prefix=/usr/local &&
make && make install' could still install files outside of /usr/local.
 No?

- Grant

[gentoo-user] Re: Installing outside of Portage & cruft removal

[Nikos Chantziaras]

Grant wrote:
> I thought running something like '.configure --prefix=/usr/local &&
> make && make install' could still install files outside of /usr/local.
>  No?


That's true, it can.  But *usually* it doesn't.

Re: [gentoo-user] Re: Installing outside of Portage & cruft removal

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

Am Montag, 16. Februar 2009 21:40:19 schrieb Nikos Chantziaras:
> Grant wrote:
> > I thought running something like '.configure --prefix=/usr/local &&
> > make && make install' could still install files outside of /usr/local.
> >  No?
>
> That's true, it can.  But *usually* it doesn't.

No, it can't. You have to tell it to so explicitely. Otherwise it wouldn't be 
possible to install software as unprivileged user.

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

[gentoo-user] Re: Installing outside of Portage & cruft removal

[Nikos Chantziaras]

Dirk Heinrichs wrote:
> Am Montag, 16. Februar 2009 21:40:19 schrieb Nikos Chantziaras:
>> Grant wrote:
>>> I thought running something like '.configure --prefix=/usr/local &&
>>> make && make install' could still install files outside of /usr/local.
>>>  No?
>> That's true, it can.  But *usually* it doesn't.
> 
> No, it can't. You have to tell it to so explicitely. Otherwise it wouldn't be 
> possible to install software as unprivileged user.

It's up the application to decide how to use prefix variable.  Most 
applications are respecting it.  But you make it sound like it's 
impossible to not respect it, which is not true.  I can write automake 
rules which completely ignore prefix.

Re: [gentoo-user] Re: Installing outside of Portage & cruft removal

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

Am Montag, 16. Februar 2009 21:54:51 schrieb Nikos Chantziaras:

> It's up the application to decide how to use prefix variable.  Most
> applications are respecting it.

Up to now, I didn't find one that doesn't. And if so, it'll receive a bug 
report right away.

> But you make it sound like it's
> impossible to not respect it, which is not true.

Well, if you don't, your package won't be spread widely until you've fixed it. 
You'll always find people who install sw as unprivileged user. If they can't 
install a package, they file a bug. Even some package managers build and 
install sw to a temporary directory as an unprivileged user to avoid messing 
up the system.

> I can write automake
> rules which completely ignore prefix.

Which in the end means the unprivileged user can't install your package. And 
even as root, I wouldn't. Nobody does this. It's a hipothetical case.

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] Re: Installing outside of Portage & cruft removal

[Grant]

>> It's up the application to decide how to use prefix variable.  Most
>> applications are respecting it.
>
> Up to now, I didn't find one that doesn't. And if so, it'll receive a bug
> report right away.
>
>> But you make it sound like it's
>> impossible to not respect it, which is not true.
>
> Well, if you don't, your package won't be spread widely until you've fixed it.
> You'll always find people who install sw as unprivileged user. If they can't
> install a package, they file a bug. Even some package managers build and
> install sw to a temporary directory as an unprivileged user to avoid messing
> up the system.
>
>> I can write automake
>> rules which completely ignore prefix.
>
> Which in the end means the unprivileged user can't install your package. And
> even as root, I wouldn't. Nobody does this. It's a hipothetical case.
>
> Bye...
>
>        Dirk

Is this the right thing to do?

$ ./configure --prefix=/usr/local && make
# make install

- Grant

Re: [gentoo-user] Re: Installing outside of Portage & cruft removal

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 225 bytes --]

Am Dienstag, 17. Februar 2009 19:22:31 schrieb Grant:

> Is this the right thing to do?
>
> $ ./configure --prefix=/usr/local && make
> # make install

Yes. However, the better way is to follow Alex' proposal.

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] Installing outside of Portage & cruft removal

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 396 bytes --]

Am Montag, 16. Februar 2009 21:31:39 schrieb Grant:

> I thought running something like '.configure --prefix=/usr/local &&
> make && make install' could still install files outside of /usr/local.
>  No?

As long as you don't specify otherwise, no. Everything will be under the 
directory specified by --prefix unless you specify exceptions, see 
"./configure --help".

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]