[gentoo-portage-dev] [PATCH 0/3] Implement a more verbose User-Agent HTTP-header

[gentoo-portage-dev] [PATCH 0/3] Implement a more verbose User-Agent HTTP-header

[Mark Kubacki]

From: W-Mark Kubacki <wmark@hurrikane.de>

Before this patch series Portage identified itself as
> Gentoo Portage
Now it will be the output of ```emerge --version```, i.e.:
> Gentoo Portage 2.1.13.7 (default/linux/amd64/13.0, gcc-4.6.2, 
> glibc-2.18, 3.9.0-rc8-mark-signed+ x86_64, 
> Intel(R) Core(TM) i7-3770T CPU @ 2.50GHz)

That longer string makes debugging easier and helps deciding
when to deploy changes on the Gentoo binhost's side.

W-Mark Kubacki (3):
  Send exact version with User-Agent HTTP-header
  Send output of ```emerge --version``` as User-Agent HTTP-header
  Add CPU model name to output of getportageversion as fifth element

 pym/_emerge/actions.py       | 4 +++-
 pym/portage/dbapi/bintree.py | 8 +++++++-
 pym/portage/util/_urlopen.py | 5 +++--
 3 files changed, 13 insertions(+), 4 deletions(-)

-- 
1.8.3.2

[gentoo-portage-dev] [PATCH 1/3] Send exact version with User-Agent HTTP-header

[Mark Kubacki]

From: W-Mark Kubacki <wmark@hurrikane.de>

Signed-off-by: W-Mark Kubacki <wmark@hurrikane.de>
---
 pym/portage/util/_urlopen.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/pym/portage/util/_urlopen.py b/pym/portage/util/_urlopen.py
index 768ccb8..798e7b4 100644
--- a/pym/portage/util/_urlopen.py
+++ b/pym/portage/util/_urlopen.py
@@ -6,6 +6,7 @@ import sys
 from datetime import datetime
 from time import mktime
 from email.utils import formatdate, parsedate
+from portage import VERSION
 
 try:
 	from urllib.request import urlopen as _urlopen
@@ -34,7 +35,7 @@ def urlopen(url, if_modified_since=None):
 		url = urllib_parse.urlunparse((parse_result.scheme, netloc, parse_result.path, parse_result.params, parse_result.query, parse_result.fragment))
 		password_manager = urllib_request.HTTPPasswordMgrWithDefaultRealm()
 		request = urllib_request.Request(url)
-		request.add_header('User-Agent', 'Gentoo Portage')
+		request.add_header('User-Agent', 'Gentoo Portage '+VERSION)
 		if if_modified_since:
 			request.add_header('If-Modified-Since', _timestamp_to_http(if_modified_since))
 		if parse_result.username is not None:
-- 
1.8.3.2

[gentoo-portage-dev] [PATCH 2/3] Send output of ```emerge --version``` as User-Agent HTTP-header

[Mark Kubacki]

From: W-Mark Kubacki <wmark@hurrikane.de>

The remote server logs will read like this:
  2001:db8::4 - - [14/Aug/2013:20:58:02 +0200]
  "GET /Packages HTTP/1.1" 304 0 "-"
  "Gentoo Portage 2.1.13.7 (default/linux/amd64/13.0, gcc-4.6.2,
   glibc-2.18, 3.9.0-rc8-mark-signed+ x86_64)"

This will enable administrators to…
• check if a machine uses the correct binhost
• decide when new server-side features can be enabled (Portage ver.)
• notify the machine's administrator of malconfigurations
  (such as wrong GCC libraries, profile, obsolete kernel)

Construction of this header happens before "Packages" are fetched
from a remote binhost.

Signed-off-by: W-Mark Kubacki <wmark@hurrikane.de>
---
 pym/portage/dbapi/bintree.py | 8 +++++++-
 pym/portage/util/_urlopen.py | 4 ++--
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/pym/portage/dbapi/bintree.py b/pym/portage/dbapi/bintree.py
index 61ac6b5..44ee8e6 100644
--- a/pym/portage/dbapi/bintree.py
+++ b/pym/portage/dbapi/bintree.py
@@ -20,6 +20,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
 	'portage.util.listdir:listdir',
 	'portage.util._urlopen:urlopen@_urlopen',
 	'portage.versions:best,catpkgsplit,catsplit,_pkg_str',
+	'_emerge.actions:load_emerge_config,getportageversion',
 )
 
 from portage.cache.mappings import slot_dict_class
@@ -892,8 +893,13 @@ class binarytree(object):
 				# Don't use urlopen for https, since it doesn't support
 				# certificate/hostname verification (bug #469888).
 				if parsed_url.scheme not in ('https',):
+					trees = load_emerge_config().trees
+					user_agent = "Gentoo "+getportageversion(self.settings["PORTDIR"], None,
+							self.settings.profile_path, self.settings["CHOST"],
+							trees[self.settings['EROOT']]["vartree"].dbapi)
 					try:
-						f = _urlopen(url, if_modified_since=local_timestamp)
+						f = _urlopen(url, user_agent=user_agent,
+							if_modified_since=local_timestamp)
 						if hasattr(f, 'headers') and f.headers.get('timestamp', ''):
 							remote_timestamp = f.headers.get('timestamp')
 					except IOError as err:
diff --git a/pym/portage/util/_urlopen.py b/pym/portage/util/_urlopen.py
index 798e7b4..9876f29 100644
--- a/pym/portage/util/_urlopen.py
+++ b/pym/portage/util/_urlopen.py
@@ -26,7 +26,7 @@ if sys.hexversion >= 0x3000000:
 #  and the file-'mtime'
 TIMESTAMP_TOLERANCE=5
 
-def urlopen(url, if_modified_since=None):
+def urlopen(url, user_agent=None, if_modified_since=None):
 	parse_result = urllib_parse.urlparse(url)
 	if parse_result.scheme not in ("http", "https"):
 		return _urlopen(url)
@@ -35,7 +35,7 @@ def urlopen(url, if_modified_since=None):
 		url = urllib_parse.urlunparse((parse_result.scheme, netloc, parse_result.path, parse_result.params, parse_result.query, parse_result.fragment))
 		password_manager = urllib_request.HTTPPasswordMgrWithDefaultRealm()
 		request = urllib_request.Request(url)
-		request.add_header('User-Agent', 'Gentoo Portage '+VERSION)
+		request.add_header('User-Agent', user_agent or 'Gentoo Portage '+VERSION)
 		if if_modified_since:
 			request.add_header('If-Modified-Since', _timestamp_to_http(if_modified_since))
 		if parse_result.username is not None:
-- 
1.8.3.2

[gentoo-portage-dev] [PATCH 3/3] Add CPU model name to output of getportageversion as fifth element

[Mark Kubacki]

From: W-Mark Kubacki <wmark@hurrikane.de>

It will read like this:
> Portage 2.1.13.7 (default/linux/amd64/13.0, gcc-4.6.2, glibc-2.18, 
> 3.9.0-rc8-mark-signed+ x86_64, Intel(R) Core(TM) i7-3770T CPU @ 2.50GHz)

That new fifth element will be the CPU model name of the host
running Portage. It is not the target architecture!

The former output is not sufficient to tell if a machine
has been downgraded (i.e. 3rd gen. Core i7 running a
configuration for x86 "Pentium Celeron") or to distinguish
between i.e. ARM CPUs ("arm5tel" could be a lot, including
incompatible instruction sets).

Signed-off-by: W-Mark Kubacki <wmark@hurrikane.de>
---
 pym/_emerge/actions.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 52ceba4..e1873cd 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -3017,9 +3017,11 @@ def getportageversion(portdir, _unused, profile, chost, vardb):
 
 	gccver = getgccversion(chost)
 	unameout=platform.release()+" "+platform.machine()
+	cpu_model_name=platform.processor()
 
 	return "Portage %s (%s, %s, %s, %s)" % \
-		(portage.VERSION, profilever, gccver, ",".join(libcver), unameout)
+		(portage.VERSION, profilever, gccver, ",".join(libcver), unameout,
+		cpu_model_name)
 
 def git_sync_timestamps(portdb, portdir):
 	"""
-- 
1.8.3.2

[gentoo-portage-dev] Re: [PATCH 3/3] Add CPU model name to output of getportageversion as fifth element

[Mark Kubacki]

[-- Attachment #1: Type: text/plain, Size: 354 bytes --]

On 2013-08-14 22:10, Mark Kubacki wrote:
>
>   	return "Portage %s (%s, %s, %s, %s)" % \
> -		(portage.VERSION, profilever, gccver, ",".join(libcver), unameout)
> +		(portage.VERSION, profilever, gccver, ",".join(libcver), unameout,
> +		cpu_model_name)

That one is missing a %s:

+ return "Portage %s (%s, %s, %s, %s, %s)" % \

-- 
Mark


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4339 bytes --]

Re: [gentoo-portage-dev] [PATCH 2/3] Send output of ```emerge --version``` as User-Agent HTTP-header

[Zac Medico]

On 08/14/2013 01:10 PM, Mark Kubacki wrote:
> @@ -892,8 +893,13 @@ class binarytree(object):
>  				# Don't use urlopen for https, since it doesn't support
>  				# certificate/hostname verification (bug #469888).
>  				if parsed_url.scheme not in ('https',):
> +					trees = load_emerge_config().trees
> +					user_agent = "Gentoo "+getportageversion(self.settings["PORTDIR"], None,
> +							self.settings.profile_path, self.settings["CHOST"],
> +							trees[self.settings['EROOT']]["vartree"].dbapi)

Generally, your patches look reasonable. However, it's a waste to call
load_emerge_config() here, since the caller has certainly loaded the
config already. I guess we could have the caller pass the result of
getportageversion() as an argument to the binarytree.populate() method.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH 3/3] Add CPU model name to output of getportageversion as fifth element

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 1863 bytes --]

On 14-08-2013 22:10:40 +0200, Mark Kubacki wrote:
> From: W-Mark Kubacki <wmark@hurrikane.de>
> 
> It will read like this:
> > Portage 2.1.13.7 (default/linux/amd64/13.0, gcc-4.6.2, glibc-2.18, 
> > 3.9.0-rc8-mark-signed+ x86_64, Intel(R) Core(TM) i7-3770T CPU @ 2.50GHz)
> 
> That new fifth element will be the CPU model name of the host
> running Portage. It is not the target architecture!

FYI:

% python3
Python 3.3.2 (default, Jul 24 2013, 11:14:02) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666-r1, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.release()
'12.4.0'
>>> platform.machine()
'x86_64'
>>> platform.processor()
'i386'
>>> 

% python
Python 3.2.3 (default, May  6 2013, 21:19:05) 
[GCC 4.7.2] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.release()
'5.11'
>>> platform.machine()
'i86pc'
>>> platform.processor()
'i386'
>>> 

% python
Python 3.3.2 (default, Jul 15 2013, 13:51:24) 
[GCC 4.7.2] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.release()
'5.10'
>>> platform.machine()
'sun4u'
>>> platform.processor()
'sparc'
>>> 

% python
Python 3.2.5 (default, Jul 15 2013, 11:37:08) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.release()
'3.8.13-gentoo'
>>> platform.machine()
'x86_64'
>>> platform.processor()
'AMD Athlon(tm) 64 X2 Dual Core Processor 3800+'
>>> 

e.g. it seems to me only on Linux it gives fancy model output.  Note
that the first and second system were running a 64-bit Python as well as
kernel.

Fabian

-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-portage-dev] [PATCH 3/3] Add CPU model name to output of getportageversion as fifth element

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

(This holds for all patches)

Please surround '+' with spaces[0].
Please ensure <79 width[1].
Please indent continuation lines properly[2].


[0]  <http://www.python.org/dev/peps/pep-0008/#id19>
[1]  <http://www.python.org/dev/peps/pep-0008/#id13>
[2]  <http://www.python.org/dev/peps/pep-0008/#id11>, look at foo =
long_function_name().
- -- 
Alexander
alexander@plaimi.net
http://plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlIMtFAACgkQRtClrXBQc7UFFgD/dhU6OgwtRqF7fZ81QbC6l//V
b+qNKFwGkiv8uyqzzUUBAK9Xc/w4iR40R61YXEySV+ybQ5GbrlvyjgHMzUCyobAs
=nYLt
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] [PATCH 3/3] Add CPU model name to output of getportageversion as fifth element

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 217 bytes --]

On Thursday 15 August 2013 06:58:24 Alexander Berntsen wrote:
> (This holds for all patches)
> 
> Please surround '+' with spaces[0].

same for assignments (not to be confused with keyword args to funcs)
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-portage-dev] [PATCH 3/3] Add CPU model name to output of getportageversion as fifth element

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 357 bytes --]

On Thursday 15 August 2013 04:00:57 Fabian Groffen wrote:
> e.g. it seems to me only on Linux it gives fancy model output.  Note
> that the first and second system were running a 64-bit Python as well as
> kernel.

output from the uname() syscall/C lib func is bare.  the `uname` program in 
Gentoo contains a patch to get useful information.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]