* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2011-10-25 16:36 Paweł Hajdan
0 siblings, 0 replies; 39+ messages in thread
From: Paweł Hajdan @ 2011-10-25 16:36 UTC (permalink / raw
To: gentoo-commits
commit: 149fe3801a3bb99caf635cfe26b31522171349d1
Author: Pawel Hajdan, Jr <phajdan.jr <AT> gentoo <DOT> org>
AuthorDate: Tue Oct 25 16:35:21 2011 +0000
Commit: Paweł Hajdan <phajdan.jr <AT> gentoo <DOT> org>
CommitDate: Tue Oct 25 16:35:55 2011 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/chromium-tools.git;a=commit;h=149fe380
Add tool to extract CVE numbers from a webpage.
Useful for filing the security bug based on release notes.
---
| 31 +++++++++++++++++++++++++++++++
1 files changed, 31 insertions(+), 0 deletions(-)
--git a/extract-cves.py b/extract-cves.py
new file mode 100755
index 0000000..8769511
--- /dev/null
+++ b/extract-cves.py
@@ -0,0 +1,31 @@
+#!/usr/bin/env python
+
+import re
+import sys
+import urllib2
+
+
+CVE_PATTERN = re.compile('CVE-(\d{4})-(\d+)')
+
+
+def main(argv):
+ response = urllib2.urlopen(argv[0])
+ cves = CVE_PATTERN.findall(response.read())
+ years = {}
+ for year, no in cves:
+ if year not in years:
+ years[year] = []
+ years[year].append(no)
+ result = []
+ for year in sorted(years.keys()):
+ nos = years[year]
+ if len(nos) == 1:
+ result.append('CVE-%s-%s' % (year, nos[0]))
+ else:
+ result.append('CVE-%s-{%s}' % (year, ','.join(sorted(nos))))
+ print ' '.join(result)
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main(sys.argv[1:]))
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2012-06-18 7:38 Paweł Hajdan
0 siblings, 0 replies; 39+ messages in thread
From: Paweł Hajdan @ 2012-06-18 7:38 UTC (permalink / raw
To: gentoo-commits
commit: 31e12833015525826359340f568b2b6fad783203
Author: Pawel Hajdan, Jr <phajdan.jr <AT> gentoo <DOT> org>
AuthorDate: Mon Jun 18 07:37:23 2012 +0000
Commit: Paweł Hajdan <phajdan.jr <AT> gentoo <DOT> org>
CommitDate: Mon Jun 18 07:37:23 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/chromium-tools.git;a=commit;h=31e12833
Add sqlite-vacuum script, bug #413295.
---
sqlite-vacuum.py | 26 ++++++++++++++++++++++++++
1 files changed, 26 insertions(+), 0 deletions(-)
diff --git a/sqlite-vacuum.py b/sqlite-vacuum.py
new file mode 100755
index 0000000..03b610a
--- /dev/null
+++ b/sqlite-vacuum.py
@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+
+import os.path
+
+import magic
+import sqlite3
+
+home = os.path.expanduser('~')
+
+try:
+ m = magic.open(magic.MAGIC_NONE)
+ m.load()
+
+ for root, dirs, files in os.walk(os.path.join(home, '.config', 'chromium')):
+ for f in files:
+ path = os.path.join(root, f)
+ magic_type = m.file(path)
+ if magic_type and 'SQLite' in magic_type:
+ try:
+ c = sqlite3.connect(path)
+ c.execute('VACUUM')
+ c.execute('REINDEX')
+ finally:
+ c.close()
+finally:
+ m.close()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2012-07-31 20:39 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2012-07-31 20:39 UTC (permalink / raw
To: gentoo-commits
commit: 1be11c73e5a2c0d854957696dabc1566a13061af
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Tue Jul 31 20:29:52 2012 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Tue Jul 31 20:29:52 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/chromium-tools.git;a=commit;h=1be11c73
Python 3 compatibility.
---
| 14 +++++++++-----
1 files changed, 9 insertions(+), 5 deletions(-)
--git a/extract-cves.py b/extract-cves.py
index 8769511..4ccfbf7 100755
--- a/extract-cves.py
+++ b/extract-cves.py
@@ -1,16 +1,20 @@
#!/usr/bin/env python
+from __future__ import print_function
+
import re
import sys
-import urllib2
-
+try:
+ from urllib.request import urlopen
+except ImportError:
+ from urllib2 import urlopen
CVE_PATTERN = re.compile('CVE-(\d{4})-(\d+)')
def main(argv):
- response = urllib2.urlopen(argv[0])
- cves = CVE_PATTERN.findall(response.read())
+ response = urlopen(argv[0])
+ cves = CVE_PATTERN.findall(str(response.read()))
years = {}
for year, no in cves:
if year not in years:
@@ -23,7 +27,7 @@ def main(argv):
result.append('CVE-%s-%s' % (year, nos[0]))
else:
result.append('CVE-%s-{%s}' % (year, ','.join(sorted(nos))))
- print ' '.join(result)
+ print(' '.join(result))
return 0
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2012-07-31 23:27 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2012-07-31 23:27 UTC (permalink / raw
To: gentoo-commits
commit: ad1d4e64ef7b780f9cb2dc3d7a4c00d599c08e83
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Tue Jul 31 23:27:38 2012 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Tue Jul 31 23:27:38 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/chromium-tools.git;a=commit;h=ad1d4e64
Add dependency information to sqlite-vaccum.py.
---
sqlite-vacuum.py | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/sqlite-vacuum.py b/sqlite-vacuum.py
index 03b610a..a63e312 100755
--- a/sqlite-vacuum.py
+++ b/sqlite-vacuum.py
@@ -1,5 +1,9 @@
#!/usr/bin/env python
+# Dependencies:
+# dev-lang/python[sqlite]
+# sys-apps/file[python]
+
import os.path
import magic
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2015-08-13 20:53 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2015-08-13 20:53 UTC (permalink / raw
To: gentoo-commits
commit: 70a26591a5872c48a140edd2256e89a88b24b9de
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Thu Aug 13 20:53:26 2015 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Thu Aug 13 20:53:26 2015 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=70a26591
Add my script for bumping google-chrome
chrome-bump | 140 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 140 insertions(+)
diff --git a/chrome-bump b/chrome-bump
new file mode 100755
index 0000000..b3f82a5
--- /dev/null
+++ b/chrome-bump
@@ -0,0 +1,140 @@
+#!/usr/bin/env python
+
+from __future__ import print_function
+
+import argparse
+from contextlib import closing
+from debian import deb822
+from glob import glob
+import os
+import os.path
+import portage
+import shutil
+import subprocess
+import sys
+try:
+ from urllib.request import urlopen
+except ImportError:
+ from urllib2 import urlopen
+
+ARCHIVE = 'https://dl.google.com/linux/chrome/deb'
+DIST = 'stable'
+COMP = 'main'
+ARCH = 'amd64'
+
+PORTDIR = portage.settings.repositories['gentoo'].location
+
+PKGMAP = {
+ 'www-client/google-chrome': {
+ '*.ebuild': ('_p', 'google-chrome-stable'),
+ },
+ 'www-client/google-chrome-beta': {
+ '*.ebuild': ('_p', 'google-chrome-beta'),
+ },
+ 'www-client/google-chrome-unstable': {
+ '*.ebuild': ('_p', 'google-chrome-unstable'),
+ },
+ 'www-plugins/chrome-binary-plugins': {
+ '*_p*.ebuild': ('_p', 'google-chrome-stable'),
+ '*_beta*.ebuild': ('_beta', 'google-chrome-beta'),
+ '*_alpha*.ebuild': ('_alpha', 'google-chrome-unstable'),
+ },
+}
+
+ARGS = None
+
+def get_deb_release(archive, dist):
+ url = '%s/dists/%s/Release' % (archive, dist)
+ with closing(urlopen(url)) as fp:
+ return deb822.Release(fp)
+
+def get_deb_packages(archive, dist, comp, arch):
+ url = '%s/dists/%s/%s/binary-%s/Packages' % (archive, dist, comp, arch)
+ with closing(urlopen(url)) as fp:
+ return list(deb822.Packages.iter_paragraphs(fp))
+
+def ebuild_pvr(pn, ebuild):
+ return ebuild[len(pn) + 1 : -7]
+
+def ebuild_pv(pn, ebuild):
+ return ebuild_pvr(pn, ebuild).split('-r')[0]
+
+def ebuild_version(pn, ebuild):
+ return ebuild_pv(pn, ebuild).split('_')[0]
+
+def new_ebuild(pn, version, sep, revision):
+ return pn + '-' + version + sep + revision + '.ebuild'
+
+def copy_ebuild(src, dest):
+ print('cp ' + src + ' ' + dest)
+ if not ARGS.dry_run:
+ shutil.copyfile(src, dest)
+ print('git add ' + dest)
+ if not ARGS.dry_run:
+ subprocess.check_call(['git', 'add', dest])
+
+def remove_ebuild(ebuild):
+ print('git rm ' + ebuild)
+ if not ARGS.dry_run:
+ subprocess.check_call(['git', 'rm', ebuild])
+
+def sync_ebuilds(pkg, debs):
+ os.chdir(os.path.join(PORTDIR, pkg))
+ pn = pkg.split('/')[1]
+ changed = False
+
+ for pattern in PKGMAP[pkg]:
+ (sep, name) = PKGMAP[pkg][pattern]
+ ebuilds = sorted(glob(pattern), reverse=True)
+
+ for deb in debs:
+ if deb['Package'] != name:
+ continue
+
+ (version, revision) = deb['Version'].split('-')
+ found = False
+ for ebuild in ebuilds:
+ if version == ebuild_version(pn, ebuild):
+ found = True
+ break
+ if not found:
+ copy_ebuild(ebuilds[0], new_ebuild(pn, version, sep, revision))
+ changed = True
+
+ for ebuild in ebuilds:
+ found = False
+ for deb in debs:
+ if deb['Package'] != name:
+ continue
+
+ (version, revision) = deb['Version'].split('-')
+ if version == ebuild_version(pn, ebuild):
+ found = True
+ break
+ if not found:
+ remove_ebuild(ebuild)
+ changed = True
+
+ if changed:
+ if ARGS.commit:
+ print('repoman commit')
+ if not ARGS.dry_run:
+ subprocess.check_call(['repoman', 'commit', '-S', '-m', pkg + ': automated update'])
+ else:
+ print('repoman manifest')
+ if not ARGS.dry_run:
+ subprocess.check_call(['repoman', 'manifest'])
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--commit', '-c', action='store_true')
+ parser.add_argument('--dry-run', '-n', action='store_true')
+ global ARGS
+ ARGS = parser.parse_args()
+
+ debs = get_deb_packages(ARCHIVE, DIST, COMP, ARCH)
+ for pkg in PKGMAP:
+ sync_ebuilds(pkg, debs)
+
+if __name__ == '__main__':
+ main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2016-09-15 16:11 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2016-09-15 16:11 UTC (permalink / raw
To: gentoo-commits
commit: f1b0fc1adfd405ff9ebb5738959ce4966371f936
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 15 16:10:49 2016 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Thu Sep 15 16:10:49 2016 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=f1b0fc1a
extract-cves: simplify output
This allows for easy copy/paste into the bugzilla alias field.
| 18 +++---------------
1 file changed, 3 insertions(+), 15 deletions(-)
--git a/extract-cves.py b/extract-cves.py
index 4ccfbf7..a1dc5ee 100755
--- a/extract-cves.py
+++ b/extract-cves.py
@@ -9,25 +9,13 @@ try:
except ImportError:
from urllib2 import urlopen
-CVE_PATTERN = re.compile('CVE-(\d{4})-(\d+)')
+CVE_PATTERN = re.compile('CVE-\d{4}-\d+')
def main(argv):
response = urlopen(argv[0])
- cves = CVE_PATTERN.findall(str(response.read()))
- years = {}
- for year, no in cves:
- if year not in years:
- years[year] = []
- years[year].append(no)
- result = []
- for year in sorted(years.keys()):
- nos = years[year]
- if len(nos) == 1:
- result.append('CVE-%s-%s' % (year, nos[0]))
- else:
- result.append('CVE-%s-{%s}' % (year, ','.join(sorted(nos))))
- print(' '.join(result))
+ cves = set(CVE_PATTERN.findall(str(response.read())))
+ print(','.join(cves))
return 0
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2016-09-15 16:15 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2016-09-15 16:15 UTC (permalink / raw
To: gentoo-commits
commit: 5a6c895eb7b1ab210533130a948235ac812e78ef
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 15 16:14:43 2016 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Thu Sep 15 16:14:43 2016 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=5a6c895e
chrome-bump: drop the version suffix from google-chrome
chrome-bump | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/chrome-bump b/chrome-bump
index b3f82a5..d8df1fc 100755
--- a/chrome-bump
+++ b/chrome-bump
@@ -26,16 +26,16 @@ PORTDIR = portage.settings.repositories['gentoo'].location
PKGMAP = {
'www-client/google-chrome': {
- '*.ebuild': ('_p', 'google-chrome-stable'),
+ '*.ebuild': ('', 'google-chrome-stable'),
},
'www-client/google-chrome-beta': {
- '*.ebuild': ('_p', 'google-chrome-beta'),
+ '*.ebuild': ('', 'google-chrome-beta'),
},
'www-client/google-chrome-unstable': {
- '*.ebuild': ('_p', 'google-chrome-unstable'),
+ '*.ebuild': ('', 'google-chrome-unstable'),
},
'www-plugins/chrome-binary-plugins': {
- '*_p*.ebuild': ('_p', 'google-chrome-stable'),
+ '*[0123456789].ebuild': ('', 'google-chrome-stable'),
'*_beta*.ebuild': ('_beta', 'google-chrome-beta'),
'*_alpha*.ebuild': ('_alpha', 'google-chrome-unstable'),
},
@@ -98,7 +98,7 @@ def sync_ebuilds(pkg, debs):
found = True
break
if not found:
- copy_ebuild(ebuilds[0], new_ebuild(pn, version, sep, revision))
+ copy_ebuild(ebuilds[0], new_ebuild(pn, version, sep, ''))
changed = True
for ebuild in ebuilds:
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2020-10-26 17:48 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2020-10-26 17:48 UTC (permalink / raw
To: gentoo-commits
commit: 1c4fc5a149952a22b74a8d0fc498716ee5c7dfa8
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Mon Oct 26 17:47:40 2020 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Mon Oct 26 17:47:40 2020 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=1c4fc5a1
Update chrome-bump script
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>
chrome-bump | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/chrome-bump b/chrome-bump
index d8df1fc..1b138e5 100755
--- a/chrome-bump
+++ b/chrome-bump
@@ -51,7 +51,7 @@ def get_deb_release(archive, dist):
def get_deb_packages(archive, dist, comp, arch):
url = '%s/dists/%s/%s/binary-%s/Packages' % (archive, dist, comp, arch)
with closing(urlopen(url)) as fp:
- return list(deb822.Packages.iter_paragraphs(fp))
+ return list(deb822.Packages.iter_paragraphs(fp, use_apt_pkg=False))
def ebuild_pvr(pn, ebuild):
return ebuild[len(pn) + 1 : -7]
@@ -82,11 +82,15 @@ def sync_ebuilds(pkg, debs):
os.chdir(os.path.join(PORTDIR, pkg))
pn = pkg.split('/')[1]
changed = False
+ new_versions = []
for pattern in PKGMAP[pkg]:
(sep, name) = PKGMAP[pkg][pattern]
ebuilds = sorted(glob(pattern), reverse=True)
+ if not ebuilds:
+ continue
+
for deb in debs:
if deb['Package'] != name:
continue
@@ -99,6 +103,7 @@ def sync_ebuilds(pkg, debs):
break
if not found:
copy_ebuild(ebuilds[0], new_ebuild(pn, version, sep, ''))
+ new_versions.append(version)
changed = True
for ebuild in ebuilds:
@@ -119,7 +124,10 @@ def sync_ebuilds(pkg, debs):
if ARGS.commit:
print('repoman commit')
if not ARGS.dry_run:
- subprocess.check_call(['repoman', 'commit', '-S', '-m', pkg + ': automated update'])
+ message = '%s: automated update (%s)' % (pkg, ', '.join(new_versions))
+ if len(message) > 69:
+ message = '%s: automated update' % pkg
+ subprocess.check_call(['repoman', 'commit', '-S', '-m', message])
else:
print('repoman manifest')
if not ARGS.dry_run:
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2020-11-21 19:34 Stephan Hartmann
0 siblings, 0 replies; 39+ messages in thread
From: Stephan Hartmann @ 2020-11-21 19:34 UTC (permalink / raw
To: gentoo-commits
commit: 7e33e3e27556dd8c7774b5b2ec94250c4e687f50
Author: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
AuthorDate: Sat Nov 21 19:34:38 2020 +0000
Commit: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
CommitDate: Sat Nov 21 19:34:38 2020 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=7e33e3e2
Add alternative chrome bump script
Signed-off-by: Stephan Hartmann <sultan <AT> gentoo.org>
bump_chrome.py | 258 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 258 insertions(+)
diff --git a/bump_chrome.py b/bump_chrome.py
new file mode 100755
index 0000000..a046815
--- /dev/null
+++ b/bump_chrome.py
@@ -0,0 +1,258 @@
+#!/bin/env python3
+
+import argparse
+import json
+import os
+import shutil
+import sys
+import urllib.request
+
+from portage.dbapi.porttree import portdbapi
+from portage.versions import *
+from portage.package.ebuild import digestgen, config
+from portage.output import EOutput
+
+from git import Repo
+
+channels = ["stable", "beta", "dev"]
+pkg_data = \
+{
+ "www-client" :
+ {
+ "stable" :
+ {
+ "pkg" : "google-chrome",
+ "suffix" : None,
+ "version" : None,
+ "bump" : False,
+ "stable" : False
+ },
+ "beta" :
+ {
+ "pkg" : "google-chrome-beta",
+ "suffix" : None,
+ "version" : None,
+ "bump" : False,
+ "stable" : False
+ },
+ "dev" :
+ {
+ "pkg" : "google-chrome-unstable",
+ "suffix" : None,
+ "version" : None,
+ "bump" : False,
+ "stable" : False
+ }
+ },
+ "www-plugins" :
+ {
+ "stable" :
+ {
+ "pkg" : "chrome-binary-plugins",
+ "suffix" : None,
+ "version" : None,
+ "bump" : False,
+ "stable" : True
+ },
+ "beta" :
+ {
+ "pkg" : "chrome-binary-plugins",
+ "suffix" : "beta",
+ "version" : None,
+ "bump" : False,
+ "stable" : False
+ },
+ "dev" :
+ {
+ "pkg" : "chrome-binary-plugins",
+ "suffix" : "alpha",
+ "version" : None,
+ "bump" : False,
+ "stable" : False
+ }
+ }
+}
+
+def getChromeVersionData(base_url, os):
+ if not base_url.endswith("/"):
+ url = base_url + "/"
+ url += f"all.json?os={os}"
+
+ response = urllib.request.urlopen(url)
+ data = json.loads(response.read())
+ return data[0]["versions"]
+
+def getChromeChannelVersion(versions, channel):
+ for item in versions:
+ if item["channel"] == channel:
+ return item["current_version"]
+ return None
+
+def isMajorBump(uversion, tversion):
+ uv_list = uversion.split('.')
+ tv_list = tversion.split('.')
+ if int(uv_list[0]) > int(tv_list[0]):
+ return True
+ return False
+
+def getPrevChannel(channel):
+ channel_list = channels + [channels[len(channels) - 1]]
+ for i in range(0, len(channel_list) - 1):
+ if channel_list[i] == channel:
+ return channel_list[i + 1]
+ raise ValueError(f"Unknown channel \"{channel}\".")
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--commit', '-c', action='store_true')
+ parser.add_argument('--dry-run', '-n', action='store_true')
+ args = parser.parse_args()
+
+ output = EOutput()
+
+ output.einfo("Fetching upstream version information ...")
+
+ versions = getChromeVersionData(base_url="https://omahaproxy.appspot.com",
+ os="linux")
+
+ chrome_info = {}
+ for channel in channels:
+ chrome_info[channel] = None
+
+ for channel in channels:
+ chrome_info[channel] = getChromeChannelVersion(versions=versions,
+ channel=channel)
+
+ output.einfo("Looking up Chrome version information in tree ...")
+
+ db = portdbapi()
+ repo_path = db.getRepositoryPath(repository_id="gentoo")
+ for category in pkg_data.keys():
+ for channel in channels:
+ pkg = pkg_data[category][channel]["pkg"]
+ cpvs = db.cp_list(mycp=f"{category}/{pkg}", mytree=repo_path)
+ pkg_data[category][channel]["version"] = None
+ for cpv in cpvs:
+ (cp, version, rev) = pkgsplit(mypkg=cpv)
+ suffix = pkg_data[category][channel]['suffix']
+ if suffix is not None:
+ suffix = "_" + suffix
+ if version.endswith(suffix):
+ pkg_data[category][channel]["version"] = version[:-len(suffix)]
+ elif not "_" in version:
+ pkg_data[category][channel]["version"] = version
+ if pkg_data[category][channel]["version"] is None:
+ output.ewarn("Couldn't determine tree version for "+
+ "{category}/{pkg}")
+
+ output.einfo("Comparing Chrome version informations ...")
+
+ for channel in channels:
+ if chrome_info[channel] is None:
+ output.ewarn(f"Upstream version unknown for channel \"{channel}\".")
+ else:
+ for category in pkg_data.keys():
+ pkg_data[category][channel]["bump"] = False
+ ver_info = vercmp(chrome_info[channel],
+ pkg_data[category][channel]["version"])
+ if ver_info is None:
+ output.ewarn("Cannot determine new version for " +
+ f"channel \"{channel}\" of " +
+ f"{category}/" +
+ f"{pkg_data[category][channel]['pkg']}.")
+ elif ver_info > 0:
+ pkg_data[category][channel]["bump"] = True
+ elif ver_info < 0:
+ output.ewarn("Upstream reverted bump for " +
+ f"channel \"{channel}\" of " +
+ f"{category}/" +
+ f"{pkg_data[category][channel]['pkg']}.")
+
+ for category in pkg_data.keys():
+ for channel in channels:
+ pkg = pkg_data[category][channel]["pkg"]
+ output.einfo(f"{category}/{pkg} version information:")
+ need_bump = pkg_data[category][channel]["bump"]
+ uversion = chrome_info[channel]
+ tversion = pkg_data[category][channel]["version"]
+ output.einfo(f"\t{channel}\t{tversion}\t{uversion}" +
+ f"\t==> {'bump' if need_bump else 'no bump'}")
+
+ repo = Repo(repo_path)
+ if repo.is_dirty():
+ output.eerror("Git Repository is dirty, can't continue.")
+ sys.exit(1)
+
+ index = repo.index
+ for channel in channels:
+ for category in pkg_data.keys():
+ if not pkg_data[category][channel]["bump"]:
+ continue
+ uversion = chrome_info[channel]
+ tversion = pkg_data[category][channel]["version"]
+ major_bump = isMajorBump(uversion=uversion, tversion=tversion)
+ pkg = pkg_data[category][channel]["pkg"]
+ suffix = pkg_data[category][channel]["suffix"]
+ if suffix is not None:
+ suffix = "_" + suffix
+ else:
+ suffix = ""
+ output.einfo(f"Bumping {category}/{pkg} ...")
+ if major_bump:
+ prev_channel = getPrevChannel(channel=channel)
+ prev_pkg = pkg_data[category][prev_channel]["pkg"]
+ prev_version = pkg_data[category][prev_channel]["version"]
+ prev_suffix = pkg_data[category][prev_channel]["suffix"]
+ print(prev_pkg)
+ if prev_suffix is not None:
+ prev_suffix = "_" + prev_suffix
+ else:
+ prev_suffix = ""
+ from_ebuild = os.path.join(repo_path,
+ category,
+ prev_pkg,
+ prev_pkg + "-" +
+ prev_version + prev_suffix +
+ ".ebuild")
+ else:
+ from_ebuild = os.path.join(repo_path,
+ category,
+ pkg,
+ pkg + "-" +
+ tversion + suffix +
+ ".ebuild")
+ to_ebuild = os.path.join(repo_path,
+ category,
+ pkg,
+ pkg + "-" +
+ uversion + suffix +
+ ".ebuild")
+
+ shutil.copyfile(from_ebuild, to_ebuild)
+
+ index.add(to_ebuild)
+ if major_bump:
+ old_ebuild = os.path.join(repo_path,
+ category,
+ pkg,
+ pkg + "-" +
+ tversion + suffix +
+ ".ebuild")
+ index.remove(old_ebuild, working_tree=True)
+ else:
+ index.remove(from_ebuild, working_tree=True)
+
+ to_path = os.path.dirname(to_ebuild)
+ cfg = config.config()
+ cfg["O"] = to_path
+
+ digestgen.digestgen(None, cfg, db)
+
+ index.add(os.path.join(to_path, "Manifest"))
+
+ repo.git.commit("-m",
+ f"{category}/{pkg}: automated update ({uversion})",
+ "-s", "-S")
+
+if __name__ == "__main__":
+ main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2022-01-31 20:20 Stephan Hartmann
0 siblings, 0 replies; 39+ messages in thread
From: Stephan Hartmann @ 2022-01-31 20:20 UTC (permalink / raw
To: gentoo-commits
commit: 715c1a649eed5da8b292bf105351024c90ba6f9e
Author: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
AuthorDate: Mon Jan 31 20:19:56 2022 +0000
Commit: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
CommitDate: Mon Jan 31 20:19:56 2022 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=715c1a64
Add edge-bump script
Signed-off-by: Stephan Hartmann <sultan <AT> gentoo.org>
edge-bump | 347 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 347 insertions(+)
diff --git a/edge-bump b/edge-bump
new file mode 100755
index 0000000..5a44177
--- /dev/null
+++ b/edge-bump
@@ -0,0 +1,347 @@
+#!/bin/env python3
+
+import argparse
+import json
+import os
+import shutil
+import sys
+import urllib.request
+import subprocess
+import functools
+import operator
+
+from debian import deb822
+from contextlib import closing
+
+from portage.dbapi.porttree import portdbapi
+from portage.versions import *
+from portage.package.ebuild import digestgen, config
+from portage.output import EOutput
+
+from git import Repo
+
+pkg_data = \
+{
+ "stable" :
+ {
+ "pkg" : "microsoft-edge",
+ "suffix" : "stable",
+ "version" : [],
+ "dversion" : [],
+ "bversion" : [],
+ "stable" : True,
+ "count" : 1
+ },
+ "beta" :
+ {
+ "pkg" : "microsoft-edge-beta",
+ "suffix" : None,
+ "version" : [],
+ "dversion" : [],
+ "bversion" : [],
+ "stable" : False,
+ "count" : 3
+ },
+ "dev" :
+ {
+ "pkg" : "microsoft-edge-dev",
+ "suffix" : None,
+ "version" : [],
+ "dversion" : [],
+ "bversion" : [],
+ "stable" : False,
+ "count" : 3
+ }
+}
+
+def getEdgeVersionData(base_url, archive, dist, comp, arch):
+ if not base_url.endswith("/"):
+ url = base_url + "/"
+ url += f"{archive}/dists/{dist}/{comp}/binary-{arch}/Packages"
+
+ with closing(urllib.request.urlopen(url)) as fp:
+ return list(deb822.Packages.iter_paragraphs(fp, use_apt_pkg=False))
+
+def compareEdgeVersion(item1, item2):
+ return -vercmp(item1[0], item2[0])
+
+def getEdgeChannelVersions(versions, channel):
+ pkg = pkg_data[channel]["pkg"]
+ if pkg_data[channel]["suffix"] is not None:
+ pkg += "-" + pkg_data[channel]["suffix"]
+ v = []
+ for item in versions:
+ if item["Package"] == pkg:
+ (version, revision) = item["Version"].split("-")
+ v.append((version, revision))
+ v.sort(key=functools.cmp_to_key(compareEdgeVersion))
+ return v
+
+def isMajorBump(uversion, tversion):
+ uv_list = uversion.split(".")
+ tv_list = tversion.split(".")
+ if int(uv_list[0]) > int(tv_list[0]):
+ return True
+ return False
+
+def getPrevChannel(channel):
+ channels = list(pkg_data.keys())
+ channel_list = channels + [channels[len(channels) - 1]]
+ for i in range(0, len(channel_list) - 1):
+ if channel_list[i] == channel:
+ return channel_list[i + 1]
+ raise ValueError(f"Unknown channel \"{channel}\".")
+
+def getEbuildVersion(version):
+ if version[1] == "r0":
+ return version[0]
+ return f"{version[0]}-{version[1]}"
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--dry-run", "-n", action="store_true")
+ args = parser.parse_args()
+
+ output = EOutput()
+
+ output.einfo("Fetching upstream version information ...")
+
+ versions = getEdgeVersionData(base_url="https://packages.microsoft.com/repos",
+ archive="edge", dist="stable", comp="main",
+ arch="amd64")
+
+ edge_info = {}
+ for channel in pkg_data.keys():
+ edge_info[channel] = []
+
+ for channel in pkg_data.keys():
+ edge_info[channel] = getEdgeChannelVersions(versions=versions,
+ channel=channel)
+
+ output.einfo("Looking up Edge versions information in tree ...")
+
+ db = portdbapi()
+ repo_path = db.getRepositoryPath(repository_id="gentoo")
+ for channel in pkg_data.keys():
+ pkg = pkg_data[channel]["pkg"]
+ cpvs = db.cp_list(mycp=f"www-client/{pkg}", mytree=repo_path)
+ for cpv in cpvs:
+ (cp, version, rev) = pkgsplit(mypkg=cpv)
+ pkg_data[channel]["version"].append((version,rev))
+ if len(pkg_data[channel]["version"]) == 0:
+ output.ewarn("Couldn't determine tree versions for "+
+ "www-client/{pkg}")
+ pkg_data[channel]["version"].sort(key=functools.cmp_to_key(compareEdgeVersion))
+
+ output.einfo("Comparing Edge version informations ...")
+
+ for channel in pkg_data.keys():
+ versions = map(operator.itemgetter(0), edge_info[channel])
+ for ver in pkg_data[channel]["version"]:
+ if ver[0] not in versions:
+ output.ewarn("Upstream dropped version " +
+ f"{ver} from channel " +
+ f"\"{channel}\" of www-client/" +
+ f"{pkg_data[channel]['pkg']}.")
+ pkg_data[channel]["dversion"].append(ver)
+
+ for channel in pkg_data.keys():
+ if len(edge_info[channel]) == 0:
+ output.ewarn(f"Upstream version unknown for channel \"{channel}\".")
+ else:
+ for uver in edge_info[channel]:
+ bump = None
+ for tver in pkg_data[channel]["version"]:
+ ver_info = vercmp(uver[0], getEbuildVersion(tver))
+ if ver_info is None:
+ output.ewarn("Cannot determine new version for " +
+ f"channel \"{channel}\" of " +
+ f"www-client/" +
+ f"{pkg_data[channel]['pkg']}.")
+ bump = False
+ break
+ elif ver_info > 0:
+ if bump is None:
+ bump = True
+ elif ver_info == 0:
+ bump = False
+ elif ver_info < 0:
+ bump = False
+ if bump:
+ pkg_data[channel]["bversion"].append((uver[0], "r0"))
+
+ if ( len(pkg_data[channel]["bversion"]) == 0 and
+ len(pkg_data[channel]["dversion"]) ==
+ len(pkg_data[channel]["version"]) ):
+ output.ewarn("Update would remove all versions " +
+ f"from tree for channel \"{channel}\" of " +
+ f"www-client/" +
+ f"{pkg_data[channel]['pkg']}.")
+ pkg_data[channel]["dversion"] = []
+ elif ( len(pkg_data[channel]["bversion"]) >=
+ pkg_data[channel]["count"] ):
+ count = pkg_data[channel]["count"]
+ pkg_data[channel]["bversion"] = \
+ pkg_data[channel]["bversion"][:count]
+ pkg_data[channel]["dversion"] = pkg_data[channel]["version"]
+ elif ( len(pkg_data[channel]["bversion"]) +
+ len(pkg_data[channel]["version"]) >
+ pkg_data[channel]["count"] ):
+ count = len(pkg_data[channel]["bversion"]) + \
+ len(pkg_data[channel]["version"]) - \
+ pkg_data[channel]["count"]
+ pkg_data[channel]["dversion"] = \
+ pkg_data[channel]["version"][-count:]
+
+ for channel in pkg_data.keys():
+ pkg = pkg_data[channel]["pkg"]
+ output.einfo(f"www-client/{pkg} version information:")
+ vstr = ""
+ for ver in reversed(pkg_data[channel]["version"]):
+ if ver in pkg_data[channel]["dversion"]:
+ vstr += f"({getEbuildVersion(ver)})\t"
+ else:
+ vstr += f"{getEbuildVersion(ver)}\t"
+ for ver in pkg_data[channel]["bversion"]:
+ vstr += f"{getEbuildVersion(ver)}*\t"
+ output.einfo(f"\t{channel}\t{vstr}")
+
+ if len(pkg_data[channel]["bversion"]) > 0:
+ output.einfo(f"\t\t==> bump")
+ elif len(pkg_data[channel]["dversion"]) > 0:
+ output.einfo(f"\t\t==> cleanup")
+ else:
+ output.einfo(f"\t\t==> unchanged")
+
+ if not args.dry_run:
+ repo = Repo(repo_path)
+ if repo.is_dirty():
+ output.eerror("Git Repository is dirty, can't continue.")
+ sys.exit(1)
+
+ index = repo.index
+
+ for channel in pkg_data.keys():
+ pkg = pkg_data[channel]["pkg"]
+ tver = pkg_data[channel]["version"][0]
+ tversion = getEbuildVersion(tver)
+ for uver in pkg_data[channel]["bversion"]:
+ uversion = getEbuildVersion(uver)
+ major_bump = isMajorBump(uversion=uver[0], tversion=tver[0])
+ output.einfo(f"Bumping www-client/{pkg}-{uversion} ...")
+ if major_bump:
+ prev_channel = getPrevChannel(channel=channel)
+ prev_pkg = pkg_data[prev_channel]["pkg"]
+ prev_version = getEbuildVersion(
+ pkg_data[prev_channel]["version"][0])
+ from_ebuild = os.path.join("www-client",
+ prev_pkg,
+ prev_pkg + "-" +
+ prev_version +
+ ".ebuild")
+ else:
+ from_ebuild = os.path.join("www-client",
+ pkg,
+ pkg + "-" +
+ tversion +
+ ".ebuild")
+ to_ebuild = os.path.join("www-client",
+ pkg,
+ pkg + "-" +
+ uversion +
+ ".ebuild")
+
+ if args.dry_run:
+ print(f"cp {from_ebuild} {to_ebuild}")
+ if pkg_data[channel]["stable"]:
+ print(f"ekeyword ~amd64 {to_ebuild}")
+ print(f"git add {to_ebuild}")
+ else:
+ to_ebuild = os.path.join(repo_path, to_ebuild)
+ from_ebuild = os.path.join(repo_path, from_ebuild)
+ shutil.copyfile(from_ebuild, to_ebuild)
+ if pkg_data[channel]["stable"]:
+ subprocess.check_call(["ekeyword", "~amd64", to_ebuild])
+ index.add(to_ebuild)
+
+ if args.dry_run:
+ print(f"git add {os.path.join('www-client', pkg, 'Manifest')}")
+ print("git commit -m",
+ f"\"www-client/{pkg}: automated bump",
+ f"({uversion})",
+ "-s -S\"")
+ else:
+ to_path = os.path.dirname(to_ebuild)
+ cfg = config.config()
+ cfg["O"] = to_path
+
+ digestgen.digestgen(None, cfg, db)
+ index.add(os.path.join(to_path, "Manifest"))
+
+ repo.git.commit("-m",
+ f"www-client/{pkg}: automated bump ({uversion})",
+ "-s", "-S")
+
+ if pkg_data[channel]["stable"]:
+ for bver in pkg_data[channel]["bversion"]:
+ bversion = getEbuildVersion(bver)
+ output.einfo(f"Stabilizing www-client/{pkg}-{bversion} ...")
+ ebuild = os.path.join("www-client",
+ pkg,
+ pkg + "-" +
+ bversion +
+ ".ebuild")
+ if args.dry_run:
+ print(f"ekeyword amd64 {ebuild}")
+ print(f"git add {os.path.join('www-client', pkg, 'Manifest')}")
+ print("git commit -m",
+ f"\"www-client/{pkg}: amd64 stable ({bversion})\" -s -S")
+ else:
+ ebuild = os.path.join(repo_path, ebuild)
+ subprocess.check_call(["ekeyword", "amd64", ebuild])
+ index.add(ebuild)
+
+ to_path = os.path.dirname(ebuild)
+ cfg = config.config()
+ cfg["O"] = to_path
+
+ digestgen.digestgen(None, cfg, db)
+ index.add(os.path.join(to_path, "Manifest"))
+
+ repo.git.commit("-m",
+ f"www-client/{pkg}: amd64 stable ({bversion})",
+ "-s", "-S")
+
+ for dver in pkg_data[channel]["dversion"]:
+ dversion = getEbuildVersion(dver)
+ output.einfo(f"Removing www-client/{pkg}-{dversion} ...")
+ rm_ebuild = os.path.join("www-client",
+ pkg,
+ pkg + "-" +
+ dversion +
+ ".ebuild")
+ if args.dry_run:
+ print(f"git rm {os.path.relpath(rm_ebuild, repo_path)}")
+ else:
+ rm_ebuild = os.path.join(repo_path, rm_ebuild)
+ index.remove(rm_ebuild, working_tree=True)
+
+ if len(pkg_data[channel]["dversion"]) > 0:
+ if args.dry_run:
+ print(f"git add {os.path.join('www-client', pkg, 'Manifest')}")
+ print("git commit -m",
+ f"\"www-client/{pkg}: remove old\" -s -S")
+ else:
+ to_path = os.path.dirname(rm_ebuild)
+ cfg = config.config()
+ cfg["O"] = to_path
+
+ digestgen.digestgen(None, cfg, db)
+ index.add(os.path.join(to_path, "Manifest"))
+
+ repo.git.commit("-m",
+ f"www-client/{pkg}: remove old",
+ "-s", "-S")
+
+if __name__ == "__main__":
+ main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2022-02-05 16:29 Stephan Hartmann
0 siblings, 0 replies; 39+ messages in thread
From: Stephan Hartmann @ 2022-02-05 16:29 UTC (permalink / raw
To: gentoo-commits
commit: 46e5883adc29ea0ae920800a449db4684004dc3a
Author: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
AuthorDate: Sat Feb 5 16:28:38 2022 +0000
Commit: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
CommitDate: Sat Feb 5 16:29:12 2022 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=46e5883a
bump_chrome.py: add revision and dry run support
Signed-off-by: Stephan Hartmann <sultan <AT> gentoo.org>
bump_chrome.py | 99 ++++++++++++++++++++++++++++++++++++++--------------------
1 file changed, 66 insertions(+), 33 deletions(-)
diff --git a/bump_chrome.py b/bump_chrome.py
index a046815..dadc059 100755
--- a/bump_chrome.py
+++ b/bump_chrome.py
@@ -6,6 +6,7 @@ import os
import shutil
import sys
import urllib.request
+import subprocess
from portage.dbapi.porttree import portdbapi
from portage.versions import *
@@ -25,7 +26,7 @@ pkg_data = \
"suffix" : None,
"version" : None,
"bump" : False,
- "stable" : False
+ "stable" : True
},
"beta" :
{
@@ -102,9 +103,13 @@ def getPrevChannel(channel):
return channel_list[i + 1]
raise ValueError(f"Unknown channel \"{channel}\".")
+def getEbuildVersion(version):
+ if version[1] == "r0":
+ return version[0]
+ return f"{version[0]}-{version[1]}"
+
def main():
parser = argparse.ArgumentParser()
- parser.add_argument('--commit', '-c', action='store_true')
parser.add_argument('--dry-run', '-n', action='store_true')
args = parser.parse_args()
@@ -134,13 +139,14 @@ def main():
pkg_data[category][channel]["version"] = None
for cpv in cpvs:
(cp, version, rev) = pkgsplit(mypkg=cpv)
- suffix = pkg_data[category][channel]['suffix']
+ suffix = pkg_data[category][channel]["suffix"]
if suffix is not None:
suffix = "_" + suffix
if version.endswith(suffix):
- pkg_data[category][channel]["version"] = version[:-len(suffix)]
+ pkg_data[category][channel]["version"] = (version[:-len(suffix)],
+ rev)
elif not "_" in version:
- pkg_data[category][channel]["version"] = version
+ pkg_data[category][channel]["version"] = (version, rev)
if pkg_data[category][channel]["version"] is None:
output.ewarn("Couldn't determine tree version for "+
"{category}/{pkg}")
@@ -154,7 +160,7 @@ def main():
for category in pkg_data.keys():
pkg_data[category][channel]["bump"] = False
ver_info = vercmp(chrome_info[channel],
- pkg_data[category][channel]["version"])
+ pkg_data[category][channel]["version"][0])
if ver_info is None:
output.ewarn("Cannot determine new version for " +
f"channel \"{channel}\" of " +
@@ -174,23 +180,26 @@ def main():
output.einfo(f"{category}/{pkg} version information:")
need_bump = pkg_data[category][channel]["bump"]
uversion = chrome_info[channel]
- tversion = pkg_data[category][channel]["version"]
+ tversion = getEbuildVersion(pkg_data[category][channel]["version"])
output.einfo(f"\t{channel}\t{tversion}\t{uversion}" +
f"\t==> {'bump' if need_bump else 'no bump'}")
- repo = Repo(repo_path)
- if repo.is_dirty():
- output.eerror("Git Repository is dirty, can't continue.")
- sys.exit(1)
+ if not args.dry_run:
+ repo = Repo(repo_path)
+ if repo.is_dirty():
+ output.eerror("Git Repository is dirty, can't continue.")
+ sys.exit(1)
+
+ index = repo.index
- index = repo.index
for channel in channels:
for category in pkg_data.keys():
if not pkg_data[category][channel]["bump"]:
continue
uversion = chrome_info[channel]
- tversion = pkg_data[category][channel]["version"]
- major_bump = isMajorBump(uversion=uversion, tversion=tversion)
+ tversion = getEbuildVersion(pkg_data[category][channel]["version"])
+ major_bump = isMajorBump(uversion=uversion,
+ tversion=pkg_data[category][channel]["version"][0])
pkg = pkg_data[category][channel]["pkg"]
suffix = pkg_data[category][channel]["suffix"]
if suffix is not None:
@@ -201,58 +210,82 @@ def main():
if major_bump:
prev_channel = getPrevChannel(channel=channel)
prev_pkg = pkg_data[category][prev_channel]["pkg"]
- prev_version = pkg_data[category][prev_channel]["version"]
+ prev_version = getEbuildVersion(pkg_data[category][prev_channel]["version"])
prev_suffix = pkg_data[category][prev_channel]["suffix"]
- print(prev_pkg)
if prev_suffix is not None:
prev_suffix = "_" + prev_suffix
else:
prev_suffix = ""
- from_ebuild = os.path.join(repo_path,
- category,
+ from_ebuild = os.path.join(category,
prev_pkg,
prev_pkg + "-" +
prev_version + prev_suffix +
".ebuild")
else:
- from_ebuild = os.path.join(repo_path,
- category,
+ from_ebuild = os.path.join(category,
pkg,
pkg + "-" +
tversion + suffix +
".ebuild")
- to_ebuild = os.path.join(repo_path,
- category,
+ to_ebuild = os.path.join(category,
pkg,
pkg + "-" +
uversion + suffix +
".ebuild")
- shutil.copyfile(from_ebuild, to_ebuild)
+ if args.dry_run:
+ print(f"cp {from_ebuild} {to_ebuild}")
+ if not major_bump:
+ print(f"git rm {from_ebuild}")
+ else:
+ from_ebuild = os.path.join(repo_path, from_ebuild)
+ shutil.copyfile(from_ebuild,
+ os.path.join(repo_path, to_ebuild))
+ if not major_bump:
+ index.remove(from_ebuild, working_tree=True)
- index.add(to_ebuild)
if major_bump:
- old_ebuild = os.path.join(repo_path,
- category,
+ old_ebuild = os.path.join(category,
pkg,
pkg + "-" +
tversion + suffix +
".ebuild")
- index.remove(old_ebuild, working_tree=True)
+ if args.dry_run:
+ print(f"git rm {old_ebuild}")
+ else:
+ index.remove(os.path.join(repo_path, old_ebuild),
+ working_tree=True)
+ if pkg_data[category][channel]["stable"]:
+ if args.dry_run:
+ print(f"ekeyword amd64 {to_ebuild}")
+ else:
+ subprocess.run(["ekeyword", "amd64",
+ os.path.join(repo_path, to_ebuild)])
+
+ if args.dry_run:
+ print(f"git add {to_ebuild}")
else:
- index.remove(from_ebuild, working_tree=True)
+ to_ebuild = os.path.join(repo_path, to_ebuild)
+ index.add(to_ebuild)
to_path = os.path.dirname(to_ebuild)
cfg = config.config()
cfg["O"] = to_path
- digestgen.digestgen(None, cfg, db)
+ if args.dry_run:
+ print(f"git add {os.path.join(to_path, 'Manifest')}")
+ print("git commit -m",
+ f"\"{category}/{pkg}: automated update",
+ f"({uversion}{suffix})",
+ "-s -S\"")
+ else:
+ digestgen.digestgen(None, cfg, db)
- index.add(os.path.join(to_path, "Manifest"))
+ index.add(os.path.join(to_path, "Manifest"))
- repo.git.commit("-m",
- f"{category}/{pkg}: automated update ({uversion})",
- "-s", "-S")
+ repo.git.commit("-m",
+ f"{category}/{pkg}: automated update ({uversion}{suffix})",
+ "-s", "-S")
if __name__ == "__main__":
main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2022-02-11 17:16 Stephan Hartmann
0 siblings, 0 replies; 39+ messages in thread
From: Stephan Hartmann @ 2022-02-11 17:16 UTC (permalink / raw
To: gentoo-commits
commit: bc93089a28f77b14ffe49280024e404247323848
Author: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
AuthorDate: Fri Feb 11 17:16:20 2022 +0000
Commit: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
CommitDate: Fri Feb 11 17:16:41 2022 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=bc93089a
edge-bump: copy metadata.xml
Signed-off-by: Stephan Hartmann <sultan <AT> gentoo.org>
edge-bump | 23 ++++++++++++++++++++---
1 file changed, 20 insertions(+), 3 deletions(-)
diff --git a/edge-bump b/edge-bump
index 5a44177..e77f87d 100755
--- a/edge-bump
+++ b/edge-bump
@@ -77,10 +77,11 @@ def getEdgeChannelVersions(versions, channel):
v.sort(key=functools.cmp_to_key(compareEdgeVersion))
return v
-def isMajorBump(uversion, tversion):
+def isMajorBump(channel, uversion, tversion):
uv_list = uversion.split(".")
tv_list = tversion.split(".")
- if int(uv_list[0]) > int(tv_list[0]):
+ if ( int(uv_list[0]) > int(tv_list[0]) and
+ getPrevChannel(channel=channel) != channel ):
return True
return False
@@ -227,7 +228,9 @@ def main():
tversion = getEbuildVersion(tver)
for uver in pkg_data[channel]["bversion"]:
uversion = getEbuildVersion(uver)
- major_bump = isMajorBump(uversion=uver[0], tversion=tver[0])
+ major_bump = isMajorBump(channel=channel,
+ uversion=uver[0],
+ tversion=tver[0])
output.einfo(f"Bumping www-client/{pkg}-{uversion} ...")
if major_bump:
prev_channel = getPrevChannel(channel=channel)
@@ -239,6 +242,12 @@ def main():
prev_pkg + "-" +
prev_version +
".ebuild")
+ from_meta = os.path.join("www-client",
+ prev_pkg,
+ "metadata.xml")
+ to_meta = os.path.join("www-client",
+ pkg,
+ "metadata.xml")
else:
from_ebuild = os.path.join("www-client",
pkg,
@@ -256,6 +265,9 @@ def main():
if pkg_data[channel]["stable"]:
print(f"ekeyword ~amd64 {to_ebuild}")
print(f"git add {to_ebuild}")
+ if major_bump:
+ print(f"cp {from_meta} {to_meta}")
+ print(f"git add {to_meta}")
else:
to_ebuild = os.path.join(repo_path, to_ebuild)
from_ebuild = os.path.join(repo_path, from_ebuild)
@@ -263,6 +275,11 @@ def main():
if pkg_data[channel]["stable"]:
subprocess.check_call(["ekeyword", "~amd64", to_ebuild])
index.add(to_ebuild)
+ if major_bump:
+ to_meta = os.path.join(repo_path, to_meta)
+ from_meta = os.path.join(repo_path, from_meta)
+ shutil.copyfile(from_meta, to_meta)
+ index.add(to_meta)
if args.dry_run:
print(f"git add {os.path.join('www-client', pkg, 'Manifest')}")
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2022-05-03 16:54 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2022-05-03 16:54 UTC (permalink / raw
To: gentoo-commits
commit: 856061d17ae4b31662b007fb9dd3e4b07775e94b
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Tue May 3 16:54:35 2022 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Tue May 3 16:54:35 2022 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=856061d1
Rename bump_chrome.py to chrome-bump
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>
bump_chrome.py => chrome-bump | 0
1 file changed, 0 insertions(+), 0 deletions(-)
diff --git a/bump_chrome.py b/chrome-bump
similarity index 100%
rename from bump_chrome.py
rename to chrome-bump
^ permalink raw reply [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2022-05-03 16:54 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2022-05-03 16:54 UTC (permalink / raw
To: gentoo-commits
commit: 681e72c7924ed57d7b73183fb07e28f4fb524919
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Tue May 3 16:53:01 2022 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Tue May 3 16:53:32 2022 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=681e72c7
Remove old chrome-bump script
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>
chrome-bump | 148 ------------------------------------------------------------
1 file changed, 148 deletions(-)
diff --git a/chrome-bump b/chrome-bump
deleted file mode 100755
index 1b138e5..0000000
--- a/chrome-bump
+++ /dev/null
@@ -1,148 +0,0 @@
-#!/usr/bin/env python
-
-from __future__ import print_function
-
-import argparse
-from contextlib import closing
-from debian import deb822
-from glob import glob
-import os
-import os.path
-import portage
-import shutil
-import subprocess
-import sys
-try:
- from urllib.request import urlopen
-except ImportError:
- from urllib2 import urlopen
-
-ARCHIVE = 'https://dl.google.com/linux/chrome/deb'
-DIST = 'stable'
-COMP = 'main'
-ARCH = 'amd64'
-
-PORTDIR = portage.settings.repositories['gentoo'].location
-
-PKGMAP = {
- 'www-client/google-chrome': {
- '*.ebuild': ('', 'google-chrome-stable'),
- },
- 'www-client/google-chrome-beta': {
- '*.ebuild': ('', 'google-chrome-beta'),
- },
- 'www-client/google-chrome-unstable': {
- '*.ebuild': ('', 'google-chrome-unstable'),
- },
- 'www-plugins/chrome-binary-plugins': {
- '*[0123456789].ebuild': ('', 'google-chrome-stable'),
- '*_beta*.ebuild': ('_beta', 'google-chrome-beta'),
- '*_alpha*.ebuild': ('_alpha', 'google-chrome-unstable'),
- },
-}
-
-ARGS = None
-
-def get_deb_release(archive, dist):
- url = '%s/dists/%s/Release' % (archive, dist)
- with closing(urlopen(url)) as fp:
- return deb822.Release(fp)
-
-def get_deb_packages(archive, dist, comp, arch):
- url = '%s/dists/%s/%s/binary-%s/Packages' % (archive, dist, comp, arch)
- with closing(urlopen(url)) as fp:
- return list(deb822.Packages.iter_paragraphs(fp, use_apt_pkg=False))
-
-def ebuild_pvr(pn, ebuild):
- return ebuild[len(pn) + 1 : -7]
-
-def ebuild_pv(pn, ebuild):
- return ebuild_pvr(pn, ebuild).split('-r')[0]
-
-def ebuild_version(pn, ebuild):
- return ebuild_pv(pn, ebuild).split('_')[0]
-
-def new_ebuild(pn, version, sep, revision):
- return pn + '-' + version + sep + revision + '.ebuild'
-
-def copy_ebuild(src, dest):
- print('cp ' + src + ' ' + dest)
- if not ARGS.dry_run:
- shutil.copyfile(src, dest)
- print('git add ' + dest)
- if not ARGS.dry_run:
- subprocess.check_call(['git', 'add', dest])
-
-def remove_ebuild(ebuild):
- print('git rm ' + ebuild)
- if not ARGS.dry_run:
- subprocess.check_call(['git', 'rm', ebuild])
-
-def sync_ebuilds(pkg, debs):
- os.chdir(os.path.join(PORTDIR, pkg))
- pn = pkg.split('/')[1]
- changed = False
- new_versions = []
-
- for pattern in PKGMAP[pkg]:
- (sep, name) = PKGMAP[pkg][pattern]
- ebuilds = sorted(glob(pattern), reverse=True)
-
- if not ebuilds:
- continue
-
- for deb in debs:
- if deb['Package'] != name:
- continue
-
- (version, revision) = deb['Version'].split('-')
- found = False
- for ebuild in ebuilds:
- if version == ebuild_version(pn, ebuild):
- found = True
- break
- if not found:
- copy_ebuild(ebuilds[0], new_ebuild(pn, version, sep, ''))
- new_versions.append(version)
- changed = True
-
- for ebuild in ebuilds:
- found = False
- for deb in debs:
- if deb['Package'] != name:
- continue
-
- (version, revision) = deb['Version'].split('-')
- if version == ebuild_version(pn, ebuild):
- found = True
- break
- if not found:
- remove_ebuild(ebuild)
- changed = True
-
- if changed:
- if ARGS.commit:
- print('repoman commit')
- if not ARGS.dry_run:
- message = '%s: automated update (%s)' % (pkg, ', '.join(new_versions))
- if len(message) > 69:
- message = '%s: automated update' % pkg
- subprocess.check_call(['repoman', 'commit', '-S', '-m', message])
- else:
- print('repoman manifest')
- if not ARGS.dry_run:
- subprocess.check_call(['repoman', 'manifest'])
-
-def main():
- parser = argparse.ArgumentParser()
- parser.add_argument('--commit', '-c', action='store_true')
- parser.add_argument('--dry-run', '-n', action='store_true')
- global ARGS
- ARGS = parser.parse_args()
-
- debs = get_deb_packages(ARCHIVE, DIST, COMP, ARCH)
- for pkg in PKGMAP:
- sync_ebuilds(pkg, debs)
-
-if __name__ == '__main__':
- main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2022-05-06 9:55 Stephan Hartmann
0 siblings, 0 replies; 39+ messages in thread
From: Stephan Hartmann @ 2022-05-06 9:55 UTC (permalink / raw
To: gentoo-commits
commit: eb9734c60aff0ea27cf030a88b4779ae613f88dd
Author: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
AuthorDate: Fri May 6 09:54:02 2022 +0000
Commit: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
CommitDate: Fri May 6 09:55:51 2022 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=eb9734c6
Add opera-bump script
Signed-off-by: Stephan Hartmann <sultan <AT> gentoo.org>
opera-bump | 397 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 397 insertions(+)
diff --git a/opera-bump b/opera-bump
new file mode 100755
index 0000000..c1e3c46
--- /dev/null
+++ b/opera-bump
@@ -0,0 +1,397 @@
+#!/bin/env python3
+
+import argparse
+import json
+import os
+import shutil
+import sys
+import urllib.request
+import subprocess
+import functools
+import operator
+
+from bs4 import BeautifulSoup
+from debian import deb822
+from contextlib import closing
+
+from portage.dbapi.porttree import portdbapi
+from portage.versions import *
+from portage.package.ebuild import digestgen, config
+from portage.output import EOutput
+
+from git import Repo
+
+pkg_data = \
+{
+ "stable" :
+ {
+ "pkg" : "opera",
+ "suffix" : "stable",
+ "version" : [],
+ "dversion" : [],
+ "bversion" : [],
+ "stable" : True,
+ "count" : 1
+ },
+ "beta" :
+ {
+ "pkg" : "opera-beta",
+ "suffix" : None,
+ "version" : [],
+ "dversion" : [],
+ "bversion" : [],
+ "stable" : False,
+ "count" : 3
+ },
+ "dev" :
+ {
+ "pkg" : "opera-developer",
+ "suffix" : None,
+ "version" : [],
+ "dversion" : [],
+ "bversion" : [],
+ "stable" : False,
+ "count" : 3
+ }
+}
+
+def getOperaVersionInfo(base_url, archive, arch, version):
+ if not base_url.endswith("/"):
+ url = base_url + "/"
+ url += f"{version}/linux"
+ try:
+ req = urllib.request.urlopen(url)
+ except urllib.error.HTTPError:
+ return None
+ soup = BeautifulSoup(req, "html.parser")
+ base_fn = f"{archive}_{version}_{arch}."
+ rpm = False
+ for node in soup.find_all("a"):
+ v = node.get("href")
+ if v.startswith(base_fn):
+ if v.endswith("rpm"):
+ rpm = True
+ elif v.endswith("deb"):
+ return (version, "0", "deb")
+ if rpm:
+ return (version, "0", "rpm")
+ return None
+
+def getOperaVersionData(base_url, package, archive, arch, tversion,
+ platform=None):
+ if not base_url.endswith("/"):
+ url = base_url + "/"
+ url += package
+ if platform is not None:
+ url += f"/{platform}"
+
+ req = urllib.request.urlopen(url)
+ soup = BeautifulSoup(req, "html.parser")
+ versions = []
+ for node in soup.find_all("a"):
+ v = node.get("href")
+ if v.endswith("/"):
+ v = v[:-1]
+ if v != "..":
+ check = False
+ for tver in tversion:
+ c = vercmp(v, tver[0])
+ if c is not None and c >= 0:
+ check = True
+ if check:
+ ver = getOperaVersionInfo(base_url=url,
+ archive=archive,
+ arch=arch,
+ version=v)
+ if ver is not None:
+ versions.append(ver)
+ return versions
+
+def compareOperaVersion(item1, item2):
+ return -vercmp(item1[0], item2[0])
+
+def isMajorBump(channel, uversion, tversion):
+ uv_list = uversion.split(".")
+ tv_list = tversion.split(".")
+ if ( int(uv_list[0]) > int(tv_list[0]) and
+ getPrevChannel(channel=channel) != channel ):
+ return True
+ return False
+
+def getPrevChannel(channel):
+ channels = list(pkg_data.keys())
+ channel_list = channels + [channels[len(channels) - 1]]
+ for i in range(0, len(channel_list) - 1):
+ if channel_list[i] == channel:
+ return channel_list[i + 1]
+ raise ValueError(f"Unknown channel \"{channel}\".")
+
+def getEbuildVersion(version):
+ if version[1] == "r0":
+ return version[0]
+ return f"{version[0]}-{version[1]}"
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--dry-run", "-n", action="store_true")
+ args = parser.parse_args()
+
+ output = EOutput()
+
+ output.einfo("Looking up Opera versions information in tree ...")
+
+ db = portdbapi()
+ repo_path = db.getRepositoryPath(repository_id="gentoo")
+ for channel in pkg_data.keys():
+ pkg = pkg_data[channel]["pkg"]
+ cpvs = db.cp_list(mycp=f"www-client/{pkg}", mytree=repo_path)
+ for cpv in cpvs:
+ (cp, version, rev) = pkgsplit(mypkg=cpv)
+ pkg_data[channel]["version"].append((version,rev))
+ if len(pkg_data[channel]["version"]) == 0:
+ output.ewarn("Couldn't determine tree versions for "+
+ "www-client/{pkg}")
+ pkg_data[channel]["version"].sort(key=functools.cmp_to_key(compareOperaVersion))
+
+ opera_info = {}
+ for channel in pkg_data.keys():
+ archive = pkg_data[channel]["pkg"]
+ platform = None
+ if pkg_data[channel]["suffix"] is not None:
+ archive += "-" + pkg_data[channel]["suffix"]
+ platform = "desktop"
+ output.einfo(f"Fetching upstream version information \"{archive}\" ...")
+ versions = getOperaVersionData(base_url="https://download1.operacdn.com/pub",
+ package=pkg_data[channel]["pkg"],
+ archive=archive, arch="amd64",
+ tversion=pkg_data[channel]["version"],
+ platform=platform)
+ versions.sort(key=functools.cmp_to_key(compareOperaVersion))
+ opera_info[channel] = versions
+
+ output.einfo("Comparing Opera version informations ...")
+
+ for channel in pkg_data.keys():
+ versions = map(operator.itemgetter(0), opera_info[channel])
+ for ver in pkg_data[channel]["version"]:
+ if ver[0] not in versions:
+ output.ewarn("Upstream dropped version " +
+ f"{ver[0]} from channel " +
+ f"\"{channel}\" of www-client/" +
+ f"{pkg_data[channel]['pkg']}.")
+ pkg_data[channel]["dversion"].append(ver)
+
+ for channel in pkg_data.keys():
+ if len(opera_info[channel]) == 0:
+ output.ewarn(f"Upstream version unknown for channel \"{channel}\".")
+ else:
+ for uver in opera_info[channel]:
+ bump = None
+ for tver in pkg_data[channel]["version"]:
+ ver_info = vercmp(uver[0], getEbuildVersion(tver))
+ if ver_info is None:
+ output.ewarn("Cannot determine new version for " +
+ f"channel \"{channel}\" of " +
+ f"www-client/" +
+ f"{pkg_data[channel]['pkg']}.")
+ bump = False
+ break
+ elif ver_info > 0:
+ if bump is None:
+ bump = True
+ elif ver_info == 0:
+ bump = False
+ elif ver_info < 0:
+ bump = False
+ if bump:
+ pkg_data[channel]["bversion"].append((uver[0], "r0"))
+
+ if ( len(pkg_data[channel]["bversion"]) == 0 and
+ len(pkg_data[channel]["dversion"]) ==
+ len(pkg_data[channel]["version"]) ):
+ output.ewarn("Update would remove all versions " +
+ f"from tree for channel \"{channel}\" of " +
+ f"www-client/" +
+ f"{pkg_data[channel]['pkg']}.")
+ pkg_data[channel]["dversion"] = []
+ elif ( len(pkg_data[channel]["bversion"]) >=
+ pkg_data[channel]["count"] ):
+ count = pkg_data[channel]["count"]
+ pkg_data[channel]["bversion"] = \
+ pkg_data[channel]["bversion"][:count]
+ pkg_data[channel]["dversion"] = pkg_data[channel]["version"]
+ elif ( len(pkg_data[channel]["bversion"]) +
+ len(pkg_data[channel]["version"]) >
+ pkg_data[channel]["count"] ):
+ count = len(pkg_data[channel]["bversion"]) + \
+ len(pkg_data[channel]["version"]) - \
+ pkg_data[channel]["count"]
+ pkg_data[channel]["dversion"] = \
+ pkg_data[channel]["version"][-count:]
+
+ for channel in pkg_data.keys():
+ pkg = pkg_data[channel]["pkg"]
+ output.einfo(f"www-client/{pkg} version information:")
+ vstr = ""
+ for ver in reversed(pkg_data[channel]["version"]):
+ if ver in pkg_data[channel]["dversion"]:
+ vstr += f"({getEbuildVersion(ver)})\t"
+ else:
+ vstr += f"{getEbuildVersion(ver)}\t"
+ for ver in pkg_data[channel]["bversion"]:
+ vstr += f"{getEbuildVersion(ver)}*\t"
+ output.einfo(f"\t{channel}\t{vstr}")
+
+ if len(pkg_data[channel]["bversion"]) > 0:
+ output.einfo(f"\t\t==> bump")
+ elif len(pkg_data[channel]["dversion"]) > 0:
+ output.einfo(f"\t\t==> cleanup")
+ else:
+ output.einfo(f"\t\t==> unchanged")
+
+ if not args.dry_run:
+ repo = Repo(repo_path)
+ if repo.is_dirty():
+ output.eerror("Git Repository is dirty, can't continue.")
+ sys.exit(1)
+
+ index = repo.index
+
+ for channel in pkg_data.keys():
+ pkg = pkg_data[channel]["pkg"]
+ tver = pkg_data[channel]["version"][0]
+ tversion = getEbuildVersion(tver)
+ for uver in pkg_data[channel]["bversion"]:
+ uversion = getEbuildVersion(uver)
+ major_bump = isMajorBump(channel=channel,
+ uversion=uver[0],
+ tversion=tver[0])
+ output.einfo(f"Bumping www-client/{pkg}-{uversion} ...")
+ if major_bump:
+ prev_channel = getPrevChannel(channel=channel)
+ prev_pkg = pkg_data[prev_channel]["pkg"]
+ prev_version = getEbuildVersion(
+ pkg_data[prev_channel]["version"][0])
+ from_ebuild = os.path.join("www-client",
+ prev_pkg,
+ prev_pkg + "-" +
+ prev_version +
+ ".ebuild")
+ from_meta = os.path.join("www-client",
+ prev_pkg,
+ "metadata.xml")
+ to_meta = os.path.join("www-client",
+ pkg,
+ "metadata.xml")
+ else:
+ from_ebuild = os.path.join("www-client",
+ pkg,
+ pkg + "-" +
+ tversion +
+ ".ebuild")
+ to_ebuild = os.path.join("www-client",
+ pkg,
+ pkg + "-" +
+ uversion +
+ ".ebuild")
+
+ if args.dry_run:
+ print(f"cp {from_ebuild} {to_ebuild}")
+ if pkg_data[channel]["stable"]:
+ print(f"ekeyword ~amd64 {to_ebuild}")
+ print(f"git add {to_ebuild}")
+ if major_bump:
+ print(f"cp {from_meta} {to_meta}")
+ print(f"git add {to_meta}")
+ else:
+ to_ebuild = os.path.join(repo_path, to_ebuild)
+ from_ebuild = os.path.join(repo_path, from_ebuild)
+ shutil.copyfile(from_ebuild, to_ebuild)
+ if pkg_data[channel]["stable"]:
+ subprocess.check_call(["ekeyword", "~amd64", to_ebuild])
+ index.add(to_ebuild)
+ if major_bump:
+ to_meta = os.path.join(repo_path, to_meta)
+ from_meta = os.path.join(repo_path, from_meta)
+
+ if args.dry_run:
+ print(f"git add {os.path.join('www-client', pkg, 'Manifest')}")
+ print("git commit -m",
+ f"\"www-client/{pkg}: automated bump",
+ f"({uversion})",
+ "-s -S\"")
+ else:
+ to_path = os.path.dirname(to_ebuild)
+ cfg = config.config()
+ cfg["O"] = to_path
+
+ digestgen.digestgen(None, cfg, db)
+ index.add(os.path.join(to_path, "Manifest"))
+
+ repo.git.commit("-m",
+ f"www-client/{pkg}: automated bump ({uversion})",
+ "-s", "-S")
+
+ if pkg_data[channel]["stable"]:
+ for bver in pkg_data[channel]["bversion"]:
+ bversion = getEbuildVersion(bver)
+ output.einfo(f"Stabilizing www-client/{pkg}-{bversion} ...")
+ ebuild = os.path.join("www-client",
+ pkg,
+ pkg + "-" +
+ bversion +
+ ".ebuild")
+ if args.dry_run:
+ print(f"ekeyword amd64 {ebuild}")
+ print(f"git add {os.path.join('www-client', pkg, 'Manifest')}")
+ print("git commit -m",
+ f"\"www-client/{pkg}: amd64 stable ({bversion})\" -s -S")
+ else:
+ ebuild = os.path.join(repo_path, ebuild)
+ subprocess.check_call(["ekeyword", "amd64", ebuild])
+ index.add(ebuild)
+
+ to_path = os.path.dirname(ebuild)
+ cfg = config.config()
+ cfg["O"] = to_path
+
+ digestgen.digestgen(None, cfg, db)
+ index.add(os.path.join(to_path, "Manifest"))
+
+ repo.git.commit("-m",
+ f"www-client/{pkg}: amd64 stable ({bversion})",
+ "-s", "-S")
+
+ for dver in pkg_data[channel]["dversion"]:
+ dversion = getEbuildVersion(dver)
+ output.einfo(f"Removing www-client/{pkg}-{dversion} ...")
+ rm_ebuild = os.path.join("www-client",
+ pkg,
+ pkg + "-" +
+ dversion +
+ ".ebuild")
+ if args.dry_run:
+ print(f"git rm {os.path.relpath(rm_ebuild, repo_path)}")
+ else:
+ rm_ebuild = os.path.join(repo_path, rm_ebuild)
+ index.remove(rm_ebuild, working_tree=True)
+
+ if len(pkg_data[channel]["dversion"]) > 0:
+ if args.dry_run:
+ print(f"git add {os.path.join('www-client', pkg, 'Manifest')}")
+ print("git commit -m",
+ f"\"www-client/{pkg}: remove old\" -s -S")
+ else:
+ to_path = os.path.dirname(rm_ebuild)
+ cfg = config.config()
+ cfg["O"] = to_path
+
+ digestgen.digestgen(None, cfg, db)
+ index.add(os.path.join(to_path, "Manifest"))
+
+ repo.git.commit("-m",
+ f"www-client/{pkg}: remove old",
+ "-s", "-S")
+
+if __name__ == "__main__":
+ main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2022-09-01 19:24 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2022-09-01 19:24 UTC (permalink / raw
To: gentoo-commits
commit: 2075fce3f64bed3df495af5aad2295f336ddb49a
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 1 19:24:03 2022 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Thu Sep 1 19:24:22 2022 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=2075fce3
Add package.sh
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>
package.sh | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 115 insertions(+)
diff --git a/package.sh b/package.sh
new file mode 100755
index 0000000..41dbd92
--- /dev/null
+++ b/package.sh
@@ -0,0 +1,115 @@
+#!/bin/bash -e
+
+MYPKG="chromium"
+MYVERSION="99.0.4844.51"
+#MYTARGETS=( "gcc_official" "gcc" "clang" "ozone" )
+MYTARGETS=( "gcc" "clang" "ozone" )
+
+MYRE2_SLOT="0/9"
+MYICU_SLOT="0/70.1"
+
+MYKEYWORD_PKGS=()
+#MYKEYWORD_PKGS+=( "dev-util/gn" )
+#MYKEYWORD_PKGS+=( "~media-video/ffmpeg-4.3.1" )
+#MYKEYWORD_PKGS+=( "dev-libs/re2:${MYRE2_SLOT}" )
+#MYKEYWORD_PKGS+=( "dev-libs/icu:${MYICU_SLOT}" )
+#MYKEYWORD_PKGS+=( "=media-libs/libwebp-1.2.2" )
+
+MYCATEGORY="www-client"
+MYPKGDIR="$(portageq pkgdir)"
+MYEBUILD="=${MYPKG}-${MYVERSION}"
+MYEMERGE_OPTS="-vuDN --with-bdeps=y --complete-graph"
+MYCCACHE_BASE="/var/tmp/ccache"
+MYCCACHE_USE=true
+MYUSE_FLAGS_BASE="-component-build -headless -custom-cflags cups hangouts js-type-check -kerberos pic proprietary-codecs pulseaudio suid -system-ffmpeg system-icu tcmalloc widevine"
+
+MYTARGET_USE_gcc_official=( "${MYCATEGORY}/${MYPKG} -wayland -screencast vaapi official" )
+MYTARGET_USE_gcc=( "${MYCATEGORY}/${MYPKG} -wayland -screencast -vaapi -official" )
+MYTARGET_USE_clang=( "${MYCATEGORY}/${MYPKG} -wayland -screencast -vaapi -official" )
+MYTARGET_USE_ozone=( "${MYCATEGORY}/${MYPKG} wayland screencast vaapi -official" )
+MYTARGET_USE_ozone+=( "x11-libs/gtk+ wayland" )
+MYTARGET_USE_ozone+=( "media-libs/mesa wayland" )
+
+MYCOMPILERS_gcc_official=( "gcc-9.3.0" )
+MYCOMPILERS_gcc=( "gcc-10.3.0" "gcc-11.2.0" )
+MYCOMPILERS_clang=( "clang-13" )
+MYCOMPILERS_ozone=( "gcc-11.2.0" )
+
+# merge ccache
+emerge ${MYEMERGE_OPTS} ccache
+if [[ ! -d "${MYCCACHE_BASE}" ]]; then
+ mkdir -p "${MYCCACHE_BASE}"
+ chown root:portage "${MYCCACHE_BASE}"
+ chmod 775 "${MYCCACHE_BASE}"
+fi
+
+# setup keywords
+rm -f "/etc/portage/package.accept_keywords/chromium"
+echo "=${MYCATEGORY}/${MYPKG}-${MYVERSION}" >> "/etc/portage/package.accept_keywords/chromium"
+
+for mypkg in "${MYKEYWORD_PKGS[@]}"; do
+ echo "$mypkg" >> "/etc/portage/package.accept_keywords/chromium"
+done
+
+# unmask chromium (dev channels are usually masked)
+rm -f "/etc/portage/package.unmask/chromium"
+echo "=${MYCATEGORY}/${MYPKG}-${MYVERSION}" >> "/etc/portage/package.unmask/chromium"
+
+# setup base USE flags
+echo "${MYCATEGORY}/${MYPKG} ${MYUSE_FLAGS_BASE}" > "/etc/portage/package.use/chromium-base"
+
+for mytarget in "${MYTARGETS[@]}"; do
+
+ # setup build USE flags
+ mytarget_use="MYTARGET_USE_${mytarget}[@]"
+ rm -f "/etc/portage/package.use/chromium-build"
+ for myuse in "${!mytarget_use}"; do
+ echo "${myuse}" >> "/etc/portage/package.use/chromium-build"
+ done
+
+ mycompilers="MYCOMPILERS_${mytarget}[@]"
+ for mycompiler in "${!mycompilers}"; do
+ mycc="x86_64-pc-linux-gnu-${mycompiler}"
+ if [[ ${mycompiler} = gcc* ]]; then
+ mycxx="x86_64-pc-linux-gnu-${mycompiler/gcc/g++}"
+ else
+ mycxx="x86_64-pc-linux-gnu-${mycompiler/clang/clang++}"
+ fi
+ rm -f "/etc/portage/env/chromium-cc.conf"
+ echo "CC=\"${mycc}\"" >> "/etc/portage/env/chromium-cc.conf"
+ echo "CXX=\"${mycxx}\"" >> "/etc/portage/env/chromium-cc.conf"
+ myversion="${mycompiler#*-}"
+ # remove installed package
+ emerge --depclean -v "${MYCATEGORY}/${MYPKG}"
+
+ mytargetdir="${mytarget}_${mycompiler%%-*}${myversion%%.*}"
+
+ # setup ccache
+ if [[ ! -d "${MYCCACHE_BASE}/${mytargetdir}" ]]; then
+ mkdir "${MYCCACHE_BASE}/${mytargetdir}"
+ chown root:portage "${MYCCACHE_BASE}/${mytargetdir}"
+ chmod 775 "${MYCCACHE_BASE}/${mytargetdir}"
+ fi
+ if [[ ! -f "${MYCCACHE_BASE}/${mytargetdir}/ccache.conf" ]]; then
+ cat <<- EOF > "${MYCCACHE_BASE}/${mytargetdir}/ccache.conf"
+ max_size = 10.0G
+ umask = 002
+ compression = false
+ EOF
+ chown root:portage "${MYCCACHE_BASE}/${mytargetdir}/ccache.conf"
+ chmod 664 "${MYCCACHE_BASE}/${mytargetdir}/ccache.conf"
+ fi
+ echo "CCACHE_DIR=\"${MYCCACHE_BASE}/${mytargetdir}\"" >> "/etc/portage/env/chromium-cc.conf"
+ if [[ ${MYCCACHE_USE} = true ]]; then
+ echo "FEATURES=\"ccache\"" >> "/etc/portage/env/chromium-cc.conf"
+ fi
+
+ # build chromium
+ emerge ${MYEMERGE_OPTS} "${MYEBUILD}"
+ # create binary package
+# PKGDIR="${MYPKGDIR}/${mytargetdir}" quickpkg ${MYPKG} --include-config=y
+ # update access rights
+ find "${MYPKGDIR}" -type d -exec chmod 755 {} \;
+ find "${MYPKGDIR}" -type f -exec chmod 644 {} \;
+ done
+done
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2022-09-01 19:33 Mike Gilbert
0 siblings, 0 replies; 39+ messages in thread
From: Mike Gilbert @ 2022-09-01 19:33 UTC (permalink / raw
To: gentoo-commits
commit: 3c17da8e49841cf2eaa474d595e79e6d8bd11093
Author: Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 1 19:33:45 2022 +0000
Commit: Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Thu Sep 1 19:33:45 2022 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=3c17da8e
Add package-beta.sh
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>
package.sh => package-beta.sh | 17 ++++++++---------
package.sh => package-stable.sh | 0
2 files changed, 8 insertions(+), 9 deletions(-)
diff --git a/package.sh b/package-beta.sh
similarity index 90%
copy from package.sh
copy to package-beta.sh
index 41dbd92..c2991c3 100755
--- a/package.sh
+++ b/package-beta.sh
@@ -1,9 +1,8 @@
#!/bin/bash -e
MYPKG="chromium"
-MYVERSION="99.0.4844.51"
-#MYTARGETS=( "gcc_official" "gcc" "clang" "ozone" )
-MYTARGETS=( "gcc" "clang" "ozone" )
+MYVERSION="105.0.5195.52"
+MYTARGETS=( "gcc_official" "gcc" "clang" "ozone" )
MYRE2_SLOT="0/9"
MYICU_SLOT="0/70.1"
@@ -11,9 +10,9 @@ MYICU_SLOT="0/70.1"
MYKEYWORD_PKGS=()
#MYKEYWORD_PKGS+=( "dev-util/gn" )
#MYKEYWORD_PKGS+=( "~media-video/ffmpeg-4.3.1" )
-#MYKEYWORD_PKGS+=( "dev-libs/re2:${MYRE2_SLOT}" )
+MYKEYWORD_PKGS+=( "dev-libs/re2:${MYRE2_SLOT}" )
#MYKEYWORD_PKGS+=( "dev-libs/icu:${MYICU_SLOT}" )
-#MYKEYWORD_PKGS+=( "=media-libs/libwebp-1.2.2" )
+MYKEYWORD_PKGS+=( "~media-libs/libwebp-1.2.4" )
MYCATEGORY="www-client"
MYPKGDIR="$(portageq pkgdir)"
@@ -31,9 +30,9 @@ MYTARGET_USE_ozone+=( "x11-libs/gtk+ wayland" )
MYTARGET_USE_ozone+=( "media-libs/mesa wayland" )
MYCOMPILERS_gcc_official=( "gcc-9.3.0" )
-MYCOMPILERS_gcc=( "gcc-10.3.0" "gcc-11.2.0" )
-MYCOMPILERS_clang=( "clang-13" )
-MYCOMPILERS_ozone=( "gcc-11.2.0" )
+MYCOMPILERS_gcc=( "gcc-11.3.0" "gcc-12.1.1" )
+MYCOMPILERS_clang=( "clang-14" )
+MYCOMPILERS_ozone=( "gcc-12.1.1" )
# merge ccache
emerge ${MYEMERGE_OPTS} ccache
@@ -107,7 +106,7 @@ for mytarget in "${MYTARGETS[@]}"; do
# build chromium
emerge ${MYEMERGE_OPTS} "${MYEBUILD}"
# create binary package
-# PKGDIR="${MYPKGDIR}/${mytargetdir}" quickpkg ${MYPKG} --include-config=y
+ PKGDIR="${MYPKGDIR}/${mytargetdir}" quickpkg ${MYPKG} --include-config=y
# update access rights
find "${MYPKGDIR}" -type d -exec chmod 755 {} \;
find "${MYPKGDIR}" -type f -exec chmod 644 {} \;
diff --git a/package.sh b/package-stable.sh
similarity index 100%
rename from package.sh
rename to package-stable.sh
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2023-02-05 15:09 Stephan Hartmann
0 siblings, 0 replies; 39+ messages in thread
From: Stephan Hartmann @ 2023-02-05 15:09 UTC (permalink / raw
To: gentoo-commits
commit: 952b6f7dbf1dd25590e0689751c2d30ef93d1664
Author: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
AuthorDate: Sun Feb 5 15:08:38 2023 +0000
Commit: Stephan Hartmann <sultan <AT> gentoo <DOT> org>
CommitDate: Sun Feb 5 15:09:11 2023 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=952b6f7d
chrome-bump: migrate away from omahaproxy
Will be shut down end of March 2023.
Signed-off-by: Stephan Hartmann <sultan <AT> gentoo.org>
chrome-bump | 22 +++++++---------------
1 file changed, 7 insertions(+), 15 deletions(-)
diff --git a/chrome-bump b/chrome-bump
index dadc059..7559a29 100755
--- a/chrome-bump
+++ b/chrome-bump
@@ -74,20 +74,13 @@ pkg_data = \
}
}
-def getChromeVersionData(base_url, os):
+def getChromeVersionData(base_url, os, channel):
if not base_url.endswith("/"):
url = base_url + "/"
- url += f"all.json?os={os}"
-
+ url += f"{os}/channels/{channel}/versions/all/releases?filter=endtime=1970-01-01T00:00:00Z"
response = urllib.request.urlopen(url)
data = json.loads(response.read())
- return data[0]["versions"]
-
-def getChromeChannelVersion(versions, channel):
- for item in versions:
- if item["channel"] == channel:
- return item["current_version"]
- return None
+ return data["releases"][0]["version"]
def isMajorBump(uversion, tversion):
uv_list = uversion.split('.')
@@ -117,16 +110,15 @@ def main():
output.einfo("Fetching upstream version information ...")
- versions = getChromeVersionData(base_url="https://omahaproxy.appspot.com",
- os="linux")
-
chrome_info = {}
for channel in channels:
chrome_info[channel] = None
for channel in channels:
- chrome_info[channel] = getChromeChannelVersion(versions=versions,
- channel=channel)
+ version = getChromeVersionData(base_url="https://versionhistory.googleapis.com/v1/chrome/platforms",
+ os="linux",
+ channel=channel)
+ chrome_info[channel] = version
output.einfo("Looking up Chrome version information in tree ...")
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-03-20 21:45 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-03-20 21:45 UTC (permalink / raw
To: gentoo-commits
commit: ac5ba1d42b4b09b67ff88af53722a52531315c7f
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Wed Mar 20 21:44:44 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Wed Mar 20 21:44:44 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=ac5ba1d4
chrome-bump: Add some hacky logic to update chromedriver-bin with chrome stable
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
chrome-bump | 51 +++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 43 insertions(+), 8 deletions(-)
diff --git a/chrome-bump b/chrome-bump
index 7559a29..fa132e8 100755
--- a/chrome-bump
+++ b/chrome-bump
@@ -71,7 +71,19 @@ pkg_data = \
"bump" : False,
"stable" : False
}
- }
+ },
+ # This will be parsed last so we can take advantage of google chrome stable queries
+ "www-apps":
+ {
+ "stable":
+ {
+ "pkg": "chromedriver-bin",
+ "suffix": None,
+ "version": None,
+ "bump": False,
+ "stable": True
+ }
+ },
}
def getChromeVersionData(base_url, os, channel):
@@ -126,6 +138,9 @@ def main():
repo_path = db.getRepositoryPath(repository_id="gentoo")
for category in pkg_data.keys():
for channel in channels:
+ # We only care about chromedriver that matches the stable version of google-chrome
+ if category == "www-apps" and channel != "stable":
+ continue
pkg = pkg_data[category][channel]["pkg"]
cpvs = db.cp_list(mycp=f"{category}/{pkg}", mytree=repo_path)
pkg_data[category][channel]["version"] = None
@@ -143,16 +158,26 @@ def main():
output.ewarn("Couldn't determine tree version for "+
"{category}/{pkg}")
- output.einfo("Comparing Chrome version informations ...")
+ output.einfo("Comparing Chrome version information...")
for channel in channels:
if chrome_info[channel] is None:
output.ewarn(f"Upstream version unknown for channel \"{channel}\".")
else:
for category in pkg_data.keys():
- pkg_data[category][channel]["bump"] = False
+ # chromedriver-bin is basically a shim for google-chrome stable for version purposes
+ if category == "www-apps":
+ if channel == "stable":
+ pkg_data[category][channel]["bump"] = pkg_data["www-client"][channel]["bump"]
+ pkg_data[category][channel]["version"] = pkg_data["www-client"][channel]["version"]
+ else:
+ continue
+
+ else:
+ pkg_data[category][channel]["bump"] = False
+
ver_info = vercmp(chrome_info[channel],
- pkg_data[category][channel]["version"][0])
+ pkg_data[category][channel]["version"][0])
if ver_info is None:
output.ewarn("Cannot determine new version for " +
f"channel \"{channel}\" of " +
@@ -168,6 +193,8 @@ def main():
for category in pkg_data.keys():
for channel in channels:
+ if category == "www-apps" and channel != "stable":
+ continue
pkg = pkg_data[category][channel]["pkg"]
output.einfo(f"{category}/{pkg} version information:")
need_bump = pkg_data[category][channel]["bump"]
@@ -186,6 +213,8 @@ def main():
for channel in channels:
for category in pkg_data.keys():
+ if category == "www-apps" and channel != "stable":
+ continue
if not pkg_data[category][channel]["bump"]:
continue
uversion = chrome_info[channel]
@@ -200,10 +229,16 @@ def main():
suffix = ""
output.einfo(f"Bumping {category}/{pkg} ...")
if major_bump:
- prev_channel = getPrevChannel(channel=channel)
- prev_pkg = pkg_data[category][prev_channel]["pkg"]
- prev_version = getEbuildVersion(pkg_data[category][prev_channel]["version"])
- prev_suffix = pkg_data[category][prev_channel]["suffix"]
+ if category != "www-apps":
+ prev_channel = getPrevChannel(channel=channel)
+ prev_pkg = pkg_data[category][prev_channel]["pkg"]
+ prev_version = getEbuildVersion(pkg_data[category][prev_channel]["version"])
+ prev_suffix = pkg_data[category][prev_channel]["suffix"]
+ else:
+ # Grab the details for google-chrome; we never have a suffix
+ prev_pkg = pkg_data["www-apps"]["stable"]["pkg"]
+ prev_version = getEbuildVersion(pkg_data["www-client"]["stable"]["version"])
+ prev_suffix = None
if prev_suffix is not None:
prev_suffix = "_" + prev_suffix
else:
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-03-20 21:45 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-03-20 21:45 UTC (permalink / raw
To: gentoo-commits
commit: 3b0f8cfc4a3d2e3344be2786a6ea9c255c43f8ee
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Wed Mar 20 21:45:00 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Wed Mar 20 21:45:00 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=3b0f8cfc
New script: get_chromium_toolchain_strings.sh
Just does some awks and seds to enable the automated update
of google toolchain information within / from the specified
ebuild
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get_chromium_toolchain_strings.sh | 70 +++++++++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
diff --git a/get_chromium_toolchain_strings.sh b/get_chromium_toolchain_strings.sh
new file mode 100755
index 0000000..483d66f
--- /dev/null
+++ b/get_chromium_toolchain_strings.sh
@@ -0,0 +1,70 @@
+#!/bin/bash
+
+# This script extracts version information from Chromium sources by way of a Gentoo ebuild
+# then plugs the version information into the ebuild file. This is useful for updating the
+# toolchain versions in the ebuild file when a new (major) version of Chromium is released.
+
+# Usage: get_chromium_toolchain_strings.sh <ebuild_file>
+# <ebuild_file>: The path to the Chromium ebuild file
+
+# Extract the version string from an ebuild
+get_version() {
+ local filename="$1"
+ [[ -z "$filename" ]] && return 1 # Check for empty filename
+ local version_match="${filename##*-}"; # Extract everything after the last hyphen
+ version_match="${version_match%.*}" # Remove extension (.ebuild)
+ echo "$version_match"
+}
+
+# Display script usage
+usage() {
+ echo "Usage: get_chromium_toolchain_strings.sh <ebuild_file>"
+ echo " <ebuild_file>: The path to the Chromium ebuild file"
+}
+
+# Get the ebuild filename as the first argument
+ebuild_file="$1"
+
+# Check for missing argument
+if [[ -z "$ebuild_file" ]]; then
+ echo "Error: Please provide an ebuild filename as an argument."
+ usage
+ exit 1
+fi
+
+# Extract version from filename
+version="$(get_version "$ebuild_file")"
+
+# Check if version extraction failed (function return code)
+if [[ $? -ne 0 ]]; then
+ echo "Error: Could not extract version from filename."
+ exit 1
+fi
+
+# Construct S string based on version
+# Bad luck if you don't use /var/tmp/portage, I guess.
+S="/var/tmp/portage/www-client/chromium-${version}/work/chromium-${version}/"
+
+# Run ebuild with clean and unpack options
+ebuild "$ebuild_file" clean unpack
+
+# No secret sauce here - it's just simpler to set the field separator to a single quote
+# and then extract the final character from the sub-revision field.
+# This is a bit of a hack, but it works for now - I haven't see upstream go past the
+# 9th sub-revision yet!
+
+llvm_version=$(awk -F"'" '
+/CLANG_REVISION =/ { revision = $2 }
+/CLANG_SUB_REVISION =/ { printf("%s-%d\n", revision, substr($1, length($1), 1)) }
+' "${S}/tools/clang/scripts/update.py")
+
+rust_version=$(awk -F"'" '
+/RUST_REVISION =/ { revision = $2 }
+/RUST_SUB_REVISION =/ { printf("%s-%d\n", revision, substr($1, length($1), 1)) }
+' "${S}/tools/rust/update_rust.py")
+
+# Substitute versions into ebuild (assuming specific locations)
+sed -i "s/GOOGLE_CLANG_VER=.*/GOOGLE_CLANG_VER=${llvm_version}/" "$ebuild_file"
+sed -i "s/GOOGLE_RUST_VER=.*/GOOGLE_RUST_VER=${rust_version}/" "$ebuild_file"
+
+echo "Successfully substituted versions into $ebuild_file"
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-03-20 21:45 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-03-20 21:45 UTC (permalink / raw
To: gentoo-commits
commit: b78a5ed24c78b4f55923164346b517eccf5e9390
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Wed Mar 20 21:44:52 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Wed Mar 20 21:44:52 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=b78a5ed2
Add script to generate new gn version and tarball from git
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
bump-gn.sh | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/bump-gn.sh b/bump-gn.sh
new file mode 100755
index 0000000..1b49138
--- /dev/null
+++ b/bump-gn.sh
@@ -0,0 +1,29 @@
+#!/bin/bash
+
+# This script actually only creates an appropriately-versioned GN tarball to assist
+# in the process of bumping the GN version. It does not actually bump the GN version
+# in the gentoo tree as we need to upload the tarball to a devspace.
+
+# Users should set the following to make xz work:
+# git config --global tar.tar.xz.command "xz -T0 -9 -c"
+
+# check if /tmp/gn exists and if so delete it
+if [ -d /tmp/gn ]; then
+ rm -rf /tmp/gn
+fi
+
+# Clone the gn repo
+git clone https://gn.googlesource.com/gn /tmp/gn
+
+pushd /tmp/gn
+
+commit=$(git describe --tags)
+pattern="([^-]*)-([^-]*)-([^-]*)-(.*)"
+[[ $commit =~ $pattern ]]
+count="${BASH_REMATCH[3]}"
+
+git archive --format=tar.xz --prefix=gn-0.${count}/ -o /tmp/gn-0.${count}.tar.xz HEAD
+
+popd
+
+echo "Tarball created at /tmp/gn-0.${count}.tar.xz"
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-03-20 21:45 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-03-20 21:45 UTC (permalink / raw
To: gentoo-commits
commit: 8e0913a8a053444ee7f68cabb2075c5efbd1e246
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Wed Mar 20 21:45:04 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Wed Mar 20 21:45:04 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=8e0913a8
new script: get-opera-version-mapping
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get-opera-version-mapping.py | 115 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 115 insertions(+)
diff --git a/get-opera-version-mapping.py b/get-opera-version-mapping.py
new file mode 100755
index 0000000..43f8f32
--- /dev/null
+++ b/get-opera-version-mapping.py
@@ -0,0 +1,115 @@
+#!/usr/bin/env python
+import requests
+from bs4 import BeautifulSoup
+
+
+def get_opera_chromium_versions(base_url, start_version, end_version):
+ """
+ Extracts Opera and Chromium versions from the given base URL with version placeholders,
+ parsing content sections for versions from start_version to end_version (inclusive).
+
+ Args:
+ base_url: The base URL for Opera changelogs with a version placeholder (e.g., "https://blogs.opera.com/desktop/changelog-for-{version}/").
+ start_version: The starting version to extract information for (inclusive).
+ end_version: The ending version to extract information for (inclusive).
+
+ Returns:
+ A dictionary mapping Opera version to Chromium version.
+ If no update is mentioned, the previous Chromium version is used.
+ For missing data or errors, "unknown" is used.
+ """
+ versions = {}
+ chromium_version = None
+
+ for version in range(start_version, end_version + 1):
+ # Fix formatting issue:
+ # OR url = base_url.format(version)
+ url = base_url.format(version)
+ print(f"Processing version {version}")
+
+ try:
+ # Set a timeout to avoid hanging requests
+ response = requests.get(url, timeout=5)
+ response.raise_for_status() # Raise exception for non-200 status codes
+
+ soup = BeautifulSoup(response.content, 'html.parser')
+ content = soup.find('div', class_='content')
+
+ # Iterate through each section starting with an H4 element
+ for section in content.find_all('h4'):
+ version_str, date_str = section.text.strip().split(' – ')
+ versions[version_str] = chromium_version
+
+ # Process all content elements (including nested ones) until the next H4
+ next_sibling = section.find_next_sibling(
+ lambda tag: tag.name is not None) # Skip text nodes
+
+ # Process content elements
+ update_found = False
+ while next_sibling and next_sibling.name != 'h4':
+ if next_sibling.name == 'ul':
+ for el in next_sibling.find_all('li'):
+ if 'Update Chromium' in el.text.strip():
+ update_found = True
+ break # Stop iterating after finding update
+
+ # Assign Chromium version only if update is found
+ if update_found:
+ chromium_version = el.text.strip().split()[-1]
+
+ next_sibling = next_sibling.find_next_sibling(
+ lambda tag: tag.name is not None) # Skip text nodes
+
+ # Handle missing Chromium version
+ if not chromium_version:
+ chromium_version = "unknown"
+
+ except requests.exceptions.RequestException as e:
+ if e.args and e.args[0] == 404:
+ print(f"Version {version} not found (404)")
+ else:
+ print(f"Error fetching data for version {version}: {e}")
+ chromium_version = None # Reset chromium_version for next iteration
+
+ except Exception as e: # Catch other unexpected exceptions
+ print(f"Unexpected error: {e}")
+ chromium_version = None # Reset chromium_version for next iteration
+
+ return versions
+
+
+def remediate_unknown_versions(versions):
+ """
+ Remediates entries with "unknown" values in the versions dictionary by
+ assuming no change from the previous known version.
+
+ Args:
+ versions: A dictionary mapping Opera version to Chromium version.
+
+ Returns:
+ The modified versions dictionary with "unknown" values replaced based on previous entries.
+ """
+ previous_version = None
+ for version, chromium_version in versions.items():
+ if chromium_version == "unknown":
+ if previous_version is not None:
+ # Update with previous version
+ versions[version] = previous_version
+ else:
+ previous_version = chromium_version # Update known version for future references
+ return versions
+
+
+# Example usage
+# Base URL with version placeholder
+base_url = "https://blogs.opera.com/desktop/changelog-for-{}/"
+opera_chromium_versions = get_opera_chromium_versions(base_url, 100, 108)
+
+opera_chromium_versions = remediate_unknown_versions(opera_chromium_versions)
+
+if opera_chromium_versions:
+ for opera_version, chromium_version in opera_chromium_versions.items():
+ print(
+ f"Opera Version: {opera_version}, Chromium Version: {chromium_version}")
+else:
+ print("Failed to extract any versions.")
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-03-28 2:39 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-03-28 2:39 UTC (permalink / raw
To: gentoo-commits
commit: 27d24e85030b0653630c93f123ccc68e310d7dc4
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Mon Mar 25 07:46:52 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Mon Mar 25 10:27:02 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=27d24e85
automate chromium-ffmpeg packaging
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get-opera-version-mapping.py | 2 +-
package-chromium-ffmpeg.py | 175 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 176 insertions(+), 1 deletion(-)
diff --git a/get-opera-version-mapping.py b/get-opera-version-mapping.py
index 43f8f32..2b515b4 100755
--- a/get-opera-version-mapping.py
+++ b/get-opera-version-mapping.py
@@ -103,7 +103,7 @@ def remediate_unknown_versions(versions):
# Example usage
# Base URL with version placeholder
base_url = "https://blogs.opera.com/desktop/changelog-for-{}/"
-opera_chromium_versions = get_opera_chromium_versions(base_url, 100, 108)
+opera_chromium_versions = get_opera_chromium_versions(base_url, 100, 110)
opera_chromium_versions = remediate_unknown_versions(opera_chromium_versions)
diff --git a/package-chromium-ffmpeg.py b/package-chromium-ffmpeg.py
new file mode 100755
index 0000000..5db694e
--- /dev/null
+++ b/package-chromium-ffmpeg.py
@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+
+import re
+import os
+import logging
+import subprocess
+import requests
+
+# Configure logging
+logging.basicConfig(
+ format='%(asctime)s - %(levelname)s - %(message)s', level=logging.INFO,
+ datefmt='%Y-%m-%d %H:%M:%S')
+
+
+def get_commit(version_url):
+ """Fetches the git hash from the Chromium ffmpeg submodule URL using requests.
+
+ Args:
+ version_url: The URL of the Chromium ffmpeg submodule for a specific version.
+
+ Returns:
+ The git commit hash found in the submodule URL, or None if not found.
+ """
+ try:
+ # Use requests.get to fetch the URL content
+ response = requests.get(version_url)
+ response.raise_for_status() # Raise exception for non-200 status codes
+
+ # Search for commit hash within the 'gitlink-detail' class (adapt if needed)
+ match = re.search(
+ r'<div class="gitlink-detail">Submodule link to (.*?) of', response.text)
+ if match:
+ return match.group(1)
+ else:
+ return None
+ except requests.exceptions.RequestException as e:
+ logging.error(f"Error: Failed to fetch URL {version_url} - {e}")
+ return None
+
+
+def archive_ffmpeg(version, commit_hash):
+ """Archives the Chromium ffmpeg repository at the specified commit hash.
+
+ Args:
+ version: The Chromium major version (e.g. 123).
+ commit_hash: The git commit hash of the desired ffmpeg revision.
+ """
+ # Base directory for ffmpeg checkout (configurable)
+ ffmpeg_dir = os.getenv("FFMPEG_TEMP_DIR", "/tmp/ffmpeg")
+ # Archive filename with version substitution
+ archive_name = f"/tmp/ffmpeg-chromium-{version}.tar.xz"
+
+ repo_uri = "https://chromium.googlesource.com/chromium/third_party/ffmpeg"
+
+ # Check if ffmpeg directory already exists
+ if os.path.exists(ffmpeg_dir):
+ # Verify remote URL matches expected repository
+ try:
+ output = subprocess.run(
+ ["git", "remote", "-v"], cwd=ffmpeg_dir, capture_output=True, check=True).stdout.decode()
+ if not re.search(repo_uri, output, re.MULTILINE):
+ logging.error(
+ f"Existing ffmpeg directory {ffmpeg_dir} points to a different remote. Please remove and re-clone.")
+ exit(1)
+ except subprocess.CalledProcessError as e:
+ logging.error(f"Error verifying remote URL: {e}")
+ exit(1)
+
+ # Update existing repository
+ try:
+ subprocess.run(["git", "pull"], cwd=ffmpeg_dir, check=True)
+ except subprocess.CalledProcessError as e:
+ logging.error(f"Error updating ffmpeg repository: {e}")
+ exit(1)
+ else:
+ # Clone the Chromium ffmpeg repository
+ try:
+ subprocess.run(
+ ["git", "clone", repo_uri, ffmpeg_dir], check=True)
+ except subprocess.CalledProcessError as e:
+ logging.error(f"Error cloning ffmpeg repository: {e}")
+ exit(1)
+
+ # Archive the ffmpeg directory with prefix and specific commit hash
+ try:
+ logging.info(
+ f"Archiving ffmpeg-chromium@{commit_hash}, this may take a moment...")
+ subprocess.run(["git", "archive", "--format=tar.xz", "-o", archive_name,
+ f"--prefix=ffmpeg-chromium-{version}/", commit_hash], cwd=ffmpeg_dir, check=True)
+ logging.info(
+ f"ffmpeg-chromium@{commit_hash} archived to {archive_name}")
+ except subprocess.CalledProcessError as e:
+ logging.error(f"Error archiving ffmpeg: {e}")
+
+
+def copy_and_update_ebuild(version, commit_hash):
+ """Copies the latest ffmpeg-chromium.ebuild and updates the COMMIT variable.
+
+ Args:
+ version: The Chromium version (e.g., 124).
+ commit_hash: The git commit hash of the desired ffmpeg revision.
+ """
+ # Target directory for ffmpeg-chromium ebuilds (configurable)
+ ebuild_dir = os.getenv("FFMPEG_EBUILD_DIR",
+ "/var/db/repos/gentoo/media-video/ffmpeg-chromium")
+ # Destination ebuild filename with version substitution
+ dest_ebuild = f"ffmpeg-chromium-{version}.ebuild"
+
+ # Find the highest version ebuild file
+ highest_version = None
+ for filename in os.listdir(ebuild_dir):
+ match = re.match(r"ffmpeg-chromium-(\d+)\.ebuild", filename)
+ if match:
+ current_version = int(match.group(1))
+ if highest_version is None or current_version > highest_version:
+ highest_version = current_version
+ highest_ebuild = os.path.join(ebuild_dir, filename)
+ # Check if a higher version ebuild exists
+ if highest_version:
+ # Copy the highest version ebuild
+ try:
+ subprocess.run(["cp", highest_ebuild,
+ os.path.join(ebuild_dir, dest_ebuild)],
+ check=True,)
+ except subprocess.CalledProcessError as e:
+ logging.error(f"Error copying ebuild file: {e}")
+ exit(1)
+
+ logging.info(
+ f"Copied ffmpeg-chromium-{highest_version}.ebuild to {dest_ebuild}"
+ )
+
+ # Update the COMMIT variable in the copied ebuild
+ with open(os.path.join(ebuild_dir, dest_ebuild), "r+") as f:
+ lines = f.readlines()
+ for i, line in enumerate(lines):
+ if line.startswith("COMMIT="):
+ lines[i] = f"COMMIT={commit_hash}\n"
+ f.seek(0)
+ f.writelines(lines)
+ logging.info(
+ f"Updated COMMIT variable in {dest_ebuild} to {commit_hash}")
+ break
+ else:
+ logging.info(
+ f"No existing ffmpeg-chromium ebuilds found in {ebuild_dir}")
+
+
+def main():
+ """Main function to handle user input and script execution."""
+ version_regex = r"^\d+\.\d+(?:\.\d+(?:\.\d+)?)?$" # Validate version format
+
+ while True:
+ version = input("Enter Chromium version (e.g., 123.0.4567.890): ")
+ if re.match(version_regex, version):
+ break
+ else:
+ print(
+ "Invalid version format. Please enter a version like X.Y.Z.W (e.g., 123.0.4567.890)")
+
+ version_url = f"https://chromium.googlesource.com/chromium/src.git/+/refs/tags/{version}/third_party/ffmpeg"
+ commit_hash = get_commit(version_url)
+ if commit_hash:
+ logging.info(
+ f"Chromium version {version} uses ffmpeg commit {commit_hash}")
+ major_version = version.split(".")[0]
+ archive_ffmpeg(major_version, commit_hash)
+ copy_and_update_ebuild(major_version, commit_hash)
+ else:
+ logging.error(
+ f"Failed to retrieve commit hash for Chromium version {version}")
+
+
+if __name__ == "__main__":
+ main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-05-31 23:02 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-05-31 23:02 UTC (permalink / raw
To: gentoo-commits
commit: 6b29e0b92cec9e0f930976bbd388be3fdf11d954
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Fri May 31 03:54:38 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Fri May 31 23:02:10 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=6b29e0b9
add script to generate libaom test tarball
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
generate-libaom-test-tarball.sh | 45 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/generate-libaom-test-tarball.sh b/generate-libaom-test-tarball.sh
new file mode 100755
index 0000000..9a0d837
--- /dev/null
+++ b/generate-libaom-test-tarball.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+# This script fetches the libaom sources, checks out the appropriate tag
+# and generates a tarball that can be placed in a devspace or other
+# web-accessible site and added to SRC_URI for a given libaom release.
+# Legacy manual instructions:
+# To update test data tarball, follow these steps:
+# 1. Clone the upstream repo and check out the relevant tag,
+# or download the release tarball
+# 2. Regular cmake configure (options don't matter here):
+# cd build && cmake ..
+# 3. Set LIBAOM_TEST_DATA_PATH to the directory you want and
+# run the "make testdata" target:
+# LIBAOM_TEST_DATA_PATH=../libaom-3.7.1-testdata make testdata
+# This will download the test data from the internet.
+# 4. Create a tarball out of that directory.
+# cd .. && tar cvaf libaom-3.7.1-testdata.tar.xz libaom-3.7.1-testdata
+
+set -e
+
+if [ -d /tmp/libaom ]; then
+ rm -rf /tmp/libaom
+fi
+
+git clone https://aomedia.googlesource.com/aom /tmp/libaom
+
+pushd /tmp/libaom
+ # Assume we're getting the latest tag if not in env;
+ # we're typically only packaging the latest version.
+ LATEST_TAG="$(git tag --sort=taggerdate | tail -1)"
+ TAG="${1:-$LATEST_TAG}"
+
+ if [ -d "/tmp/libaom-${TAG:1}-testdata" ]; then
+ rm -rf "/tmp/libaom-${TAG:1}-testdata"
+ fi
+
+ echo "Packaging libaom testdata for ${TAG}"
+ git checkout "tags/${TAG}"
+
+ cd build && cmake ..
+ LIBAOM_TEST_DATA_PATH="/tmp/libaom-${TAG:1}-testdata" make -j$(nproc) testdata
+popd
+pushd /tmp
+ XZ_OPT="-T0 -9" tar cvaf "libaom-${TAG:1}-testdata.tar.xz" "libaom-${TAG:1}-testdata"
+popd
+
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-06-01 7:22 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-06-01 7:22 UTC (permalink / raw
To: gentoo-commits
commit: c3d1c029fab2ec382edeae05d0940e66406e6b0d
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Sat Jun 1 07:20:54 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Sat Jun 1 07:21:53 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=c3d1c029
add script to generate libvpx test tarball
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
generate-libvpx-test-tarball.sh | 42 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
diff --git a/generate-libvpx-test-tarball.sh b/generate-libvpx-test-tarball.sh
new file mode 100755
index 0000000..f8a3844
--- /dev/null
+++ b/generate-libvpx-test-tarball.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+# This script fetches the libvpx sources, checks out the appropriate tag
+# and generates a tarball that can be placed in a devspace or other
+# web-accessible site and added to SRC_URI for a given libvpx release.
+# Legacy manual instructions:
+# To create a new testdata tarball:
+# 1. Unpack source tarball or checkout git tag
+# 2. mkdir libvpx-testdata
+# 3. export LIBVPX_TEST_DATA_PATH=libvpx-testdata
+# 4. ./configure --enable-unit-tests --enable-vp9-highbitdepth
+# 5. make testdata
+# 6. tar -caf libvpx-testdata-${MY_PV}.tar.xz libvpx-testdata
+
+set -e
+
+if [ -d /tmp/libvpx ]; then
+ rm -rf /tmp/libvpx
+fi
+
+git clone https://github.com/webmproject/libvpx.git /tmp/libvpx
+
+pushd /tmp/libvpx
+ # Assume we're getting the latest tag if not in env;
+ # we're typically only packaging the latest version.
+ LATEST_TAG="$(git tag --sort=taggerdate | tail -1)"
+ TAG="${1:-$LATEST_TAG}"
+
+ if [ -d "/tmp/libvpx-${TAG:1}-testdata" ]; then
+ rm -rf "/tmp/libvpx-${TAG:1}-testdata"
+ fi
+
+ mkdir -p "/tmp/libvpx-${TAG:1}-testdata"
+
+ echo "Packaging libvpx testdata for ${TAG}"
+ git checkout "tags/${TAG}"
+
+ ./configure --enable-unit-tests --enable-vp9-highbitdepth
+ LIBVPX_TEST_DATA_PATH="/tmp/libvpx-${TAG:1}-testdata" make -j$(nproc) testdata
+popd
+pushd /tmp
+ XZ_OPT="-T0 -9" tar cvaf "libvpx-${TAG:1}-testdata.tar.xz" "libvpx-${TAG:1}-testdata"
+popd
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-08-30 3:39 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-08-30 3:39 UTC (permalink / raw
To: gentoo-commits
commit: 9775b00690b10139b4e1055585b543d17d91d6bf
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Fri Aug 30 03:32:47 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Fri Aug 30 03:39:38 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=9775b006
Replace chromium toolchain script with a faster one using web requests
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get-chromium-toolchain-strings.py | 63 +++++++++++++++++++++++++++++++++++
get-opera-version-mapping.py | 2 +-
get_chromium_toolchain_strings.sh | 70 ---------------------------------------
3 files changed, 64 insertions(+), 71 deletions(-)
diff --git a/get-chromium-toolchain-strings.py b/get-chromium-toolchain-strings.py
new file mode 100755
index 0000000..0582912
--- /dev/null
+++ b/get-chromium-toolchain-strings.py
@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+# This script extracts the revision and sub-revision from the update.py and update_rust.py files in the Chromium source code.
+# The revision and sub-revision are used to identify the version of Clang and Rust used in the Chromium toolchain.
+
+
+import requests
+import sys
+
+
+def get_revision_info(url):
+ """
+ Extracts revision and sub-revision from a Chromium source file URL.
+
+ Args:
+ url (str): The URL of the source file on GitHub's raw endpoint.
+
+ Returns:
+ tuple: A tuple containing the revision (str) and sub-revision (int),
+ or (None, None) if not found.
+ """
+ response = requests.get(url)
+ if response.status_code == 200:
+ text = response.content.decode('utf-8') # Decode to UTF-8
+ lines = text.splitlines()
+ revision = None
+ sub_revision = None
+ for line in lines:
+ if line.startswith("CLANG_REVISION") and not line.startswith("PACKAGE_VERSION"):
+ revision = line.split("=")[1].strip().strip("'")
+ elif line.startswith("CLANG_SUB_REVISION") and not line.startswith("PACKAGE_VERSION"):
+ sub_revision = int(line.split("=")[1].strip())
+ elif line.startswith("RUST_REVISION") and not line.startswith("specieid") and not line.startswith(" return"):
+ # I know that's spelt wrong, but apparently google cant't spell
+ revision = line.split("=")[1].strip().strip("'")
+ elif line.startswith("RUST_SUB_REVISION") and not line.startswith("specieid") and not line.startswith(" return"):
+ sub_revision = int(line.split("=")[1].strip()[-1])
+ if revision is None or sub_revision is None:
+ raise ValueError("Failed to extract revision and sub-revision")
+ return revision, sub_revision
+ else:
+ raise ValueError(f"Failed to get revision info. Status code: {response.status_code}")
+
+
+def main():
+ version = sys.argv[1] if len(sys.argv) > 1 else "128.0.6613.113"
+ # It's a lot easier to use GH raw URLs for this
+ base_url = "https://raw.githubusercontent.com/chromium/chromium/"
+ clang_url = f"{base_url}{version}/tools/clang/scripts/update.py"
+ rust_url = f"{base_url}{version}/tools/rust/update_rust.py"
+ clang_revision, clang_sub_revision = get_revision_info(clang_url)
+ rust_revision, rust_sub_revision = get_revision_info(rust_url)
+ if clang_revision and clang_sub_revision:
+ print(f"clang revision: {clang_revision}-{clang_sub_revision}")
+ else:
+ print("clang revision not found")
+ if rust_revision and rust_sub_revision:
+ print(f"rust revision: {rust_revision}-{rust_sub_revision}")
+ else:
+ print("rust revision not found")
+
+if __name__ == "__main__":
+ main()
diff --git a/get-opera-version-mapping.py b/get-opera-version-mapping.py
index 2b515b4..3e68ec9 100755
--- a/get-opera-version-mapping.py
+++ b/get-opera-version-mapping.py
@@ -103,7 +103,7 @@ def remediate_unknown_versions(versions):
# Example usage
# Base URL with version placeholder
base_url = "https://blogs.opera.com/desktop/changelog-for-{}/"
-opera_chromium_versions = get_opera_chromium_versions(base_url, 100, 110)
+opera_chromium_versions = get_opera_chromium_versions(base_url, 110, 115)
opera_chromium_versions = remediate_unknown_versions(opera_chromium_versions)
diff --git a/get_chromium_toolchain_strings.sh b/get_chromium_toolchain_strings.sh
deleted file mode 100755
index 483d66f..0000000
--- a/get_chromium_toolchain_strings.sh
+++ /dev/null
@@ -1,70 +0,0 @@
-#!/bin/bash
-
-# This script extracts version information from Chromium sources by way of a Gentoo ebuild
-# then plugs the version information into the ebuild file. This is useful for updating the
-# toolchain versions in the ebuild file when a new (major) version of Chromium is released.
-
-# Usage: get_chromium_toolchain_strings.sh <ebuild_file>
-# <ebuild_file>: The path to the Chromium ebuild file
-
-# Extract the version string from an ebuild
-get_version() {
- local filename="$1"
- [[ -z "$filename" ]] && return 1 # Check for empty filename
- local version_match="${filename##*-}"; # Extract everything after the last hyphen
- version_match="${version_match%.*}" # Remove extension (.ebuild)
- echo "$version_match"
-}
-
-# Display script usage
-usage() {
- echo "Usage: get_chromium_toolchain_strings.sh <ebuild_file>"
- echo " <ebuild_file>: The path to the Chromium ebuild file"
-}
-
-# Get the ebuild filename as the first argument
-ebuild_file="$1"
-
-# Check for missing argument
-if [[ -z "$ebuild_file" ]]; then
- echo "Error: Please provide an ebuild filename as an argument."
- usage
- exit 1
-fi
-
-# Extract version from filename
-version="$(get_version "$ebuild_file")"
-
-# Check if version extraction failed (function return code)
-if [[ $? -ne 0 ]]; then
- echo "Error: Could not extract version from filename."
- exit 1
-fi
-
-# Construct S string based on version
-# Bad luck if you don't use /var/tmp/portage, I guess.
-S="/var/tmp/portage/www-client/chromium-${version}/work/chromium-${version}/"
-
-# Run ebuild with clean and unpack options
-ebuild "$ebuild_file" clean unpack
-
-# No secret sauce here - it's just simpler to set the field separator to a single quote
-# and then extract the final character from the sub-revision field.
-# This is a bit of a hack, but it works for now - I haven't see upstream go past the
-# 9th sub-revision yet!
-
-llvm_version=$(awk -F"'" '
-/CLANG_REVISION =/ { revision = $2 }
-/CLANG_SUB_REVISION =/ { printf("%s-%d\n", revision, substr($1, length($1), 1)) }
-' "${S}/tools/clang/scripts/update.py")
-
-rust_version=$(awk -F"'" '
-/RUST_REVISION =/ { revision = $2 }
-/RUST_SUB_REVISION =/ { printf("%s-%d\n", revision, substr($1, length($1), 1)) }
-' "${S}/tools/rust/update_rust.py")
-
-# Substitute versions into ebuild (assuming specific locations)
-sed -i "s/GOOGLE_CLANG_VER=.*/GOOGLE_CLANG_VER=${llvm_version}/" "$ebuild_file"
-sed -i "s/GOOGLE_RUST_VER=.*/GOOGLE_RUST_VER=${rust_version}/" "$ebuild_file"
-
-echo "Successfully substituted versions into $ebuild_file"
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-09-26 2:36 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-09-26 2:36 UTC (permalink / raw
To: gentoo-commits
commit: df1cdd52ddf34c0c0650169b34ea332b23aae90e
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 26 02:34:18 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Thu Sep 26 02:36:24 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=df1cdd52
get-edge-cves.py: new script
This script currently grabs the current month's CVRF from MS
and filters for Edge (Chromium-based) CVEs, then does some
magick to identify the version that it was fixed in if the
API is for some reason deficient...
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get-edge-cves.py | 161 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 161 insertions(+)
diff --git a/get-edge-cves.py b/get-edge-cves.py
new file mode 100755
index 0000000..0761b4e
--- /dev/null
+++ b/get-edge-cves.py
@@ -0,0 +1,161 @@
+#!/usr/bin/env python
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+# This script extracts the Chromium version mapping for Microsoft Edge based on a given CVE ID.
+# It uses the Microsoft Security Response Center (MSRC) API to get the Common Vulnerability Reporting Framework (CVRF)
+# for a given month and extracts the Chromium version mapping for Microsoft Edge (Chromium-based) from the CVRF.
+
+# API Docs https://api.msrc.microsoft.com/cvrf/v3.0/swagger/v3/swagger.json
+
+# We can use the CVRF API to get the Common Vulnerability Reporting Framework (CVRF) for a given month.
+# We can query the API via CVE ID to get the CVRF for a specific CVE, but that just leads us back to querying
+# the month. Stretch goal to ingest directly from bgo ticket aliases and confirm the month & version?
+# https://api.msrc.microsoft.com/cvrf/v3.0/updates/CVE-2024-7969
+
+# https://api.msrc.microsoft.com/cvrf/v3.0/cvrf/2024-Aug
+# is the URL for the CVRF for August 2024
+
+# The XML looks like this:
+# <cvrfdoc
+# . . .
+# <vuln:Vulnerability
+# Ordinal="261">
+# <vuln:Title>Chromium: CVE-2024-7969 Type Confusion in V8</vuln:Title>
+# . . .
+# <vuln:ProductStatuses>
+# <vuln:Status
+# Type="Known Affected">
+# <vuln:ProductID>11655</vuln:ProductID>
+# . . .
+# </vuln:ProductStatuses>
+# . . .
+# <vuln:CVE>CVE-2024-7969</vuln:CVE>
+# . . .
+# <vuln:Remediations>
+# <vuln:Remediation
+# Type="Vendor Fix">
+# <vuln:Description>Release Notes</vuln:Description>
+# <vuln:URL />
+# <vuln:ProductID>11655</vuln:ProductID>
+# <vuln:AffectedFiles />
+# <vuln:RestartRequired>No</vuln:RestartRequired>
+# <vuln:SubType>Security Update</vuln:SubType>
+# <vuln:FixedBuild>128.0.2739.42</vuln:FixedBuild>
+# . . .
+# </vuln:Remediations>
+# . . .
+# </vuln:Vulnerability>
+
+# Process: Pick a month, get the CVRF for that month, then iterate over vulnerabilities to find the ones
+# that are for Microsoft Edge (Chromium-based) `<vuln:ProductID>11655</vuln:ProductID>`.
+# Extract the <vuln:CVE>CVE-2024-7969</vuln:CVE> to extract a CVE ID and
+# map to Chromium versions using the <vuln:FixedBuild>128.0.2739.42</vuln:FixedBuild> tag
+
+# It would be possible to parse the </vuln:Note> tag which contains a table with the chromium and edge versions
+# but each row looks like this: <td>128.0.6613.84/.85</td> which is not very useful, and we already
+# know which version of Chromium a CVE got fixed in.
+
+import datetime
+import requests
+from portage import versions as portage_versions
+import sys
+import xml.etree.ElementTree as ET
+from bs4 import BeautifulSoup
+
+
+class EdgeCVE:
+ def __init__(self, cve, title, fixedbuild):
+ self.cve: str = cve
+ self.title: str = title
+ self.fixedbuild: str | None = fixedbuild
+
+ def __str__(self):
+ return f"{self.cve}: {self.title}: Fixed {self.fixedbuild if not None else 'unknown'}"
+
+
+def get_edge_cves(year, month):
+ msrcapi = f"https://api.msrc.microsoft.com/cvrf/v3.0/cvrf/{year}-{month}"
+
+ # Get the CVRF for the specified month
+ response = requests.get(msrcapi)
+
+ if response.status_code != 200:
+ print(f"Website returned {response.status_code}")
+ print(f"Failed to get CVRF for {year}-{month}")
+ sys.exit(1)
+
+ # Parse the XML
+ root = ET.fromstring(response.text)
+
+ # Find all the vulnerabilities
+ vulnerabilities = root.findall(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}Vulnerability")
+
+ edge_cves = [] # Store the edge cves here
+ for vulnerability in vulnerabilities:
+ productstatuses = vulnerability.findall(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}ProductStatuses")
+ for productstatus in productstatuses:
+ productid = productstatus.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}ProductID")
+ if productid.text == "11655":
+ # This is a Microsoft Edge (Chromium-based) vulnerability
+ cve_id = vulnerability.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}CVE").text
+ cve_title = vulnerability.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}Title").text
+ remediations = vulnerability.findall(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}Remediations")
+ for remediation in remediations:
+ fixedbuild = remediation.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}FixedBuild")
+ if fixedbuild is not None:
+ edge_cves.append(
+ EdgeCVE(cve_id,
+ cve_title,
+ fixedbuild.text)
+ )
+ else:
+ # Fall back to parsing that horrible, horrible table in the notes
+ notes = vulnerability.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}Notes")
+ # There appear to be multiple notes, but only one has content that we want:
+ # <vuln:Note Title="FAQ" Type="FAQ" Ordinal="10"><p><strong>What is the version information for this release?</strong></p>
+ found = False
+ for note in notes:
+ if note.attrib['Title'] == "FAQ" and note.attrib['Type'] == "FAQ":
+
+ # The note contains a table with the chromium and edge versions, written in "HTML"
+ # <td>8/22/2024</td>
+ content = note.text
+
+ soup = BeautifulSoup(content, 'html.parser')
+
+ # Now you can work with the parsed HTML
+
+ # Find all the table rows
+ rows = soup.find_all('tr')
+ # We want the second row, second cell
+ if len(rows) > 1:
+ cells = rows[1].find_all('td')
+ if len(cells) > 1:
+ # We want the second cell (The first is the channel, the third the chromium version it's based on)
+ edge_version = cells[1].text
+ if portage_versions.ververify(edge_version):
+ found = True
+ edge_cves.append(
+ EdgeCVE(cve_id,
+ cve_title,
+ edge_version)
+ )
+
+ if not found:
+ edge_cves.append(
+ EdgeCVE(cve_id,
+ cve_title,
+ None)
+ )
+
+ return edge_cves
+
+
+now = datetime.datetime.now()
+year = now.year
+month = now.strftime("%B")[0:3]
+
+# Call the function with current year and month
+edge_cves = get_edge_cves(year, month)
+for cve in edge_cves:
+ print(cve)
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-09-26 2:39 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-09-26 2:39 UTC (permalink / raw
To: gentoo-commits
commit: e965e5c6d8692a7ec00d255c08f670111dee1c02
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 26 02:34:18 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Thu Sep 26 02:39:01 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=e965e5c6
get-edge-cves.py: new script
This script currently grabs the current month's CVRF from MS
and filters for Edge (Chromium-based) CVEs, then does some
magick to identify the version that it was fixed in if the
API is for some reason deficient...
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get-edge-cves.py | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 153 insertions(+)
diff --git a/get-edge-cves.py b/get-edge-cves.py
new file mode 100755
index 0000000..a41187e
--- /dev/null
+++ b/get-edge-cves.py
@@ -0,0 +1,153 @@
+#!/usr/bin/env python
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+# This script extracts the Chromium version mapping for Microsoft Edge based on a given CVE ID.
+# It uses the Microsoft Security Response Center (MSRC) API to get the Common Vulnerability Reporting Framework (CVRF)
+# for a given month and extracts the Chromium version mapping for Microsoft Edge (Chromium-based) from the CVRF.
+
+# API Docs https://api.msrc.microsoft.com/cvrf/v3.0/swagger/v3/swagger.json
+
+# We can use the CVRF API to get the Common Vulnerability Reporting Framework (CVRF) for a given month.
+# We can query the API via CVE ID to get the CVRF for a specific CVE, but that just leads us back to querying
+# the month. Stretch goal to ingest directly from bgo ticket aliases and confirm the month & version?
+# https://api.msrc.microsoft.com/cvrf/v3.0/updates/CVE-2024-7969
+
+# https://api.msrc.microsoft.com/cvrf/v3.0/cvrf/2024-Aug
+# is the URL for the CVRF for August 2024
+
+# The XML looks like this:
+# <cvrfdoc
+# . . .
+# <vuln:Vulnerability
+# Ordinal="261">
+# <vuln:Title>Chromium: CVE-2024-7969 Type Confusion in V8</vuln:Title>
+# . . .
+# <vuln:ProductStatuses>
+# <vuln:Status
+# Type="Known Affected">
+# <vuln:ProductID>11655</vuln:ProductID>
+# . . .
+# </vuln:ProductStatuses>
+# . . .
+# <vuln:CVE>CVE-2024-7969</vuln:CVE>
+# . . .
+# <vuln:Remediations>
+# <vuln:Remediation
+# Type="Vendor Fix">
+# <vuln:Description>Release Notes</vuln:Description>
+# <vuln:URL />
+# <vuln:ProductID>11655</vuln:ProductID>
+# <vuln:AffectedFiles />
+# <vuln:RestartRequired>No</vuln:RestartRequired>
+# <vuln:SubType>Security Update</vuln:SubType>
+# <vuln:FixedBuild>128.0.2739.42</vuln:FixedBuild>
+# . . .
+# </vuln:Remediations>
+# . . .
+# </vuln:Vulnerability>
+
+# Process: Pick a month, get the CVRF for that month, then iterate over vulnerabilities to find the ones
+# that are for Microsoft Edge (Chromium-based) `<vuln:ProductID>11655</vuln:ProductID>`.
+# Extract the <vuln:CVE>CVE-2024-7969</vuln:CVE> to extract a CVE ID and
+# map to Chromium versions using the <vuln:FixedBuild>128.0.2739.42</vuln:FixedBuild> tag (or the notes if we _have_ to).
+
+import datetime
+import requests
+from portage import versions as portage_versions
+import sys
+import xml.etree.ElementTree as ET
+from bs4 import BeautifulSoup
+
+
+class EdgeCVE:
+ def __init__(self, cve, title, fixedbuild):
+ self.cve: str = cve
+ self.title: str = title
+ self.fixedbuild: str | None = fixedbuild
+
+ def __str__(self):
+ return f"{self.cve}: {self.title}: Fixed {self.fixedbuild if not None else 'unknown'}"
+
+
+def get_edge_cves(year, month) -> list[EdgeCVE]:
+ msrcapi = f"https://api.msrc.microsoft.com/cvrf/v3.0/cvrf/{year}-{month}"
+
+ # Get the CVRF for the specified month
+ response = requests.get(msrcapi)
+
+ if response.status_code != 200:
+ print(f"Website returned {response.status_code}")
+ print(f"Failed to get CVRF for {year}-{month}")
+ sys.exit(1)
+
+ # Parse the XML
+ root = ET.fromstring(response.text)
+
+ # Find all the vulnerabilities
+ vulnerabilities = root.findall(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}Vulnerability")
+
+ edge_cves = [] # Store the edge cves here
+ for vulnerability in vulnerabilities:
+ productstatuses = vulnerability.findall(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}ProductStatuses")
+ for productstatus in productstatuses:
+ productid = productstatus.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}ProductID")
+ if productid.text == "11655":
+ # This is a Microsoft Edge (Chromium-based) vulnerability
+ cve_id = vulnerability.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}CVE").text
+ cve_title = vulnerability.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}Title").text
+ remediations = vulnerability.findall(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}Remediations")
+ for remediation in remediations:
+ fixedbuild = remediation.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}FixedBuild")
+ if fixedbuild is not None:
+ edge_cves.append(
+ EdgeCVE(cve_id,
+ cve_title,
+ fixedbuild.text)
+ )
+ else:
+ # Fall back to parsing that horrible, horrible table in the notes
+ notes = vulnerability.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}Notes")
+ # There appear to be multiple notes, but only one has content that we want:
+ # <vuln:Note Title="FAQ" Type="FAQ" Ordinal="10"><p><strong>What is the version information for this release?</strong></p>
+ found = False
+ for note in notes:
+ if note.attrib['Title'] == "FAQ" and note.attrib['Type'] == "FAQ":
+
+ # The note contains a table with the chromium and edge versions, written in "HTML"
+ # <td>8/22/2024</td>
+ content = note.text
+
+ soup = BeautifulSoup(content, 'html.parser')
+ rows = soup.find_all('tr')
+ # We want the second row, second cell
+ if len(rows) > 1:
+ cells = rows[1].find_all('td')
+ if len(cells) > 1:
+ # We want the second cell (The first is the channel, the third the chromium version it's based on)
+ edge_version = cells[1].text
+ if portage_versions.ververify(edge_version):
+ found = True
+ edge_cves.append(
+ EdgeCVE(cve_id,
+ cve_title,
+ edge_version)
+ )
+
+ if not found:
+ edge_cves.append(
+ EdgeCVE(cve_id,
+ cve_title,
+ None)
+ )
+
+ return edge_cves
+
+
+now = datetime.datetime.now()
+year = now.year
+month = now.strftime("%B")[0:3]
+
+# Call the function with current year and month
+edge_cves = get_edge_cves(year, month)
+for cve in edge_cves:
+ print(cve)
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-09-26 3:03 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-09-26 3:03 UTC (permalink / raw
To: gentoo-commits
commit: ef84e5951d8d040c9695bda1dc846b37ed3941d4
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 26 02:59:10 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Thu Sep 26 02:59:10 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=ef84e595
get-edge-cves.py: Refactor to tidy up
Eli couldn't stand the terribleness and tidied up a few things.
Co-authored-by: Eli Schwartz <ztrawhsce <AT> gentoo.org>
Signed-off-by: Matt Jolly <Kangie <AT> gentoo.org>
get-edge-cves.py | 31 ++++++++++++-------------------
1 file changed, 12 insertions(+), 19 deletions(-)
diff --git a/get-edge-cves.py b/get-edge-cves.py
index a41187e..4911667 100755
--- a/get-edge-cves.py
+++ b/get-edge-cves.py
@@ -51,19 +51,19 @@
# Extract the <vuln:CVE>CVE-2024-7969</vuln:CVE> to extract a CVE ID and
# map to Chromium versions using the <vuln:FixedBuild>128.0.2739.42</vuln:FixedBuild> tag (or the notes if we _have_ to).
-import datetime
-import requests
-from portage import versions as portage_versions
-import sys
+import dataclasses, datetime, sys
import xml.etree.ElementTree as ET
+
from bs4 import BeautifulSoup
+from portage import versions as portage_versions
+import requests
+@dataclasses.dataclass
class EdgeCVE:
- def __init__(self, cve, title, fixedbuild):
- self.cve: str = cve
- self.title: str = title
- self.fixedbuild: str | None = fixedbuild
+ cve: str
+ title: str
+ fixedbuild: str | None
def __str__(self):
return f"{self.cve}: {self.title}: Fixed {self.fixedbuild if not None else 'unknown'}"
@@ -100,9 +100,7 @@ def get_edge_cves(year, month) -> list[EdgeCVE]:
fixedbuild = remediation.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}FixedBuild")
if fixedbuild is not None:
edge_cves.append(
- EdgeCVE(cve_id,
- cve_title,
- fixedbuild.text)
+ EdgeCVE(cve_id, cve_title, fixedbuild.text)
)
else:
# Fall back to parsing that horrible, horrible table in the notes
@@ -128,17 +126,13 @@ def get_edge_cves(year, month) -> list[EdgeCVE]:
if portage_versions.ververify(edge_version):
found = True
edge_cves.append(
- EdgeCVE(cve_id,
- cve_title,
- edge_version)
+ EdgeCVE(cve_id, cve_title, edge_version)
)
if not found:
edge_cves.append(
- EdgeCVE(cve_id,
- cve_title,
- None)
- )
+ EdgeCVE(cve_id, cve_title, None)
+ )
return edge_cves
@@ -147,7 +141,6 @@ now = datetime.datetime.now()
year = now.year
month = now.strftime("%B")[0:3]
-# Call the function with current year and month
edge_cves = get_edge_cves(year, month)
for cve in edge_cves:
print(cve)
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-09-26 5:21 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-09-26 5:21 UTC (permalink / raw
To: gentoo-commits
commit: 95970bcad40a5a7affb82b12b7706d3abb5e39c0
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 26 05:21:06 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Thu Sep 26 05:21:06 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=95970bca
get-edge-cves.py
Add the ability to query by gentoo bug or by CVE ID.
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
.gitignore | 1 +
get-edge-cves.py | 107 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
2 files changed, 100 insertions(+), 8 deletions(-)
diff --git a/.gitignore b/.gitignore
index b057d7f..61be068 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
MANIFEST
*.pyc
+bugzilla_api_key
diff --git a/get-edge-cves.py b/get-edge-cves.py
index 4911667..8336e0f 100755
--- a/get-edge-cves.py
+++ b/get-edge-cves.py
@@ -51,12 +51,12 @@
# Extract the <vuln:CVE>CVE-2024-7969</vuln:CVE> to extract a CVE ID and
# map to Chromium versions using the <vuln:FixedBuild>128.0.2739.42</vuln:FixedBuild> tag (or the notes if we _have_ to).
-import dataclasses, datetime, sys
+import argparse, calendar, dataclasses, datetime, os, sys
import xml.etree.ElementTree as ET
from bs4 import BeautifulSoup
from portage import versions as portage_versions
-import requests
+import bugzilla, requests
@dataclasses.dataclass
@@ -136,11 +136,102 @@ def get_edge_cves(year, month) -> list[EdgeCVE]:
return edge_cves
+def get_cve_from_bug_alias(bugnumber: int) -> list[str]:
+ """
+ Queries the Gentoo bugzilla instance for the list of CVEs associated with a given bug.
-now = datetime.datetime.now()
-year = now.year
-month = now.strftime("%B")[0:3]
+ Since we, by convention, alias bugs to CVEs, we can just query the alias field.
-edge_cves = get_edge_cves(year, month)
-for cve in edge_cves:
- print(cve)
+ Args:
+ bugnumber (int): The bug number to query.
+
+ Returns:
+ list[str]: A list of CVEs associated with the bug.s
+
+ """
+ url = "bugs.gentoo.org"
+ keyfile = open(os.path.abspath('./bugzilla_api_key'))
+ api_key = keyfile.read().replace('\n','')
+ print('connecting to b.g.o')
+ bzapi = bugzilla.Bugzilla(url, api_key)
+ bug = bzapi.getbug(bugnumber)
+ cves = bug.alias
+ print(f'Bug: {bug} has {len(cves)} CVEs.')
+
+ return cves
+
+
+def get_msrc_for_cve(cve: str) -> str:
+ """
+ Do a simple webrquest to get the CVRF for a given CVE.
+
+ Args:
+ cve (str): The CVE to query.
+
+ Returns:
+ str: The CVRF for the CVE.
+ """
+
+ msrcapi = f"https://api.msrc.microsoft.com/cvrf/v3.0/updates/{cve}"
+ response = requests.get(msrcapi)
+
+ if response.status_code != 200:
+ print(f"Website returned {response.status_code}")
+ print(f"Failed to get CVRF for {cve}")
+ sys.exit(1)
+
+ # This is JSON, we want { "value": [ { "ID": "2024-Aug" }, ] }
+ return response.json().get('value')[0].get('ID')
+
+
+def parse_arguments():
+ parser = argparse.ArgumentParser(description="Script to get Edge CVEs.")
+ parser.add_argument('-m', '--month', type=int, help='Month as a number (1-12)', default=datetime.datetime.now().month)
+ parser.add_argument('-y', '--year', type=int, help='Year as a four-digit number', default=datetime.datetime.now().year)
+ parser.add_argument('-b', '--bug', nargs='*', help='List of bug identifiers')
+ parser.add_argument('-c', '--cve', nargs='*', help='List of CVE identifiers')
+ return parser.parse_args()
+
+
+def main():
+ args = parse_arguments()
+
+ if not args.bug and not args.cve:
+ month = calendar.month_name[args.month][0:3]
+ for cve in get_edge_cves(args.year, month):
+ print(cve)
+
+ elif args.bug:
+ for bug in args.bug:
+ cves = get_cve_from_bug_alias(bug)
+
+ msrcs = []
+ for cve in cves:
+ msrcs.append(get_msrc_for_cve(cve))
+
+ # Dedupe
+ msrcs = list(set(msrcs))
+
+ for msrc in msrcs:
+ for cve in get_edge_cves(msrc.split('-')[0], msrc.split('-')[1]):
+ if cve.cve in cves:
+ print(cve)
+
+ elif args.cve:
+ msrcs = []
+ cves = []
+ for cve_id in args.cve:
+ cves.append(cve_id)
+ msrcs.append(get_msrc_for_cve(cve_id))
+
+ # Dedupe
+ msrcs = list(set(msrcs))
+
+ for msrc in msrcs:
+ for cve in get_edge_cves(msrc.split('-')[0], msrc.split('-')[1]):
+ if cve.cve in cves:
+ print(cve)
+
+
+if __name__ == "__main__":
+ main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-09-26 5:29 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-09-26 5:29 UTC (permalink / raw
To: gentoo-commits
commit: 0eaf9f5b47082574caaa96e4d9adb40cc1a3f26f
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 26 05:21:06 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Thu Sep 26 05:29:26 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=0eaf9f5b
get-edge-cves.py: New functionality
Add the ability to query by gentoo bug or by CVE ID.
Or multiples thereof, or pick a specific month/year to query.
The possibilities are endless. Endless!
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
.gitignore | 1 +
get-edge-cves.py | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
2 files changed, 101 insertions(+), 8 deletions(-)
diff --git a/.gitignore b/.gitignore
index b057d7f..61be068 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
MANIFEST
*.pyc
+bugzilla_api_key
diff --git a/get-edge-cves.py b/get-edge-cves.py
index 4911667..72e68f8 100755
--- a/get-edge-cves.py
+++ b/get-edge-cves.py
@@ -51,12 +51,12 @@
# Extract the <vuln:CVE>CVE-2024-7969</vuln:CVE> to extract a CVE ID and
# map to Chromium versions using the <vuln:FixedBuild>128.0.2739.42</vuln:FixedBuild> tag (or the notes if we _have_ to).
-import dataclasses, datetime, sys
+import argparse, calendar, dataclasses, datetime, os, sys
import xml.etree.ElementTree as ET
from bs4 import BeautifulSoup
from portage import versions as portage_versions
-import requests
+import bugzilla, requests
@dataclasses.dataclass
@@ -137,10 +137,102 @@ def get_edge_cves(year, month) -> list[EdgeCVE]:
return edge_cves
-now = datetime.datetime.now()
-year = now.year
-month = now.strftime("%B")[0:3]
+def get_cve_from_bug_alias(bugnumber: int) -> list[str]:
+ """
+ Queries the Gentoo bugzilla instance for the list of CVEs associated with a given bug.
-edge_cves = get_edge_cves(year, month)
-for cve in edge_cves:
- print(cve)
+ Since we, by convention, alias bugs to CVEs, we can just query the alias field.
+
+ Args:
+ bugnumber (int): The bug number to query.
+
+ Returns:
+ list[str]: A list of CVEs associated with the bug.s
+
+ """
+ url = "bugs.gentoo.org"
+ keyfile = open(os.path.abspath('./bugzilla_api_key'))
+ api_key = keyfile.read().replace('\n','')
+ print('connecting to b.g.o')
+ bzapi = bugzilla.Bugzilla(url, api_key)
+ bug = bzapi.getbug(bugnumber)
+ cves = bug.alias
+ print(f'Bug: {bug} has {len(cves)} CVEs.')
+
+ return cves
+
+
+def get_msrc_for_cve(cve: str) -> str:
+ """
+ Do a simple webrquest to get the CVRF for a given CVE.
+
+ Args:
+ cve (str): The CVE to query.
+
+ Returns:
+ str: The CVRF for the CVE.
+ """
+
+ msrcapi = f"https://api.msrc.microsoft.com/cvrf/v3.0/updates/{cve}"
+ response = requests.get(msrcapi)
+
+ if response.status_code != 200:
+ print(f"Website returned {response.status_code}")
+ print(f"Failed to get CVRF for {cve}")
+ sys.exit(1)
+
+ # This is JSON, we want { "value": [ { "ID": "2024-Aug" }, ] }
+ return response.json().get('value')[0].get('ID')
+
+
+def parse_arguments():
+ parser = argparse.ArgumentParser(description="Script to get Edge CVEs.")
+ parser.add_argument('-m', '--month', type=int, help='Month as a number (1-12)', default=datetime.datetime.now().month)
+ parser.add_argument('-y', '--year', type=int, help='Year as a four-digit number', default=datetime.datetime.now().year)
+ parser.add_argument('-b', '--bug', nargs='*', help='List of bug identifiers')
+ parser.add_argument('-c', '--cve', nargs='*', help='List of CVE identifiers')
+ return parser.parse_args()
+
+
+def main():
+ args = parse_arguments()
+
+ if not args.bug and not args.cve:
+ month = calendar.month_name[args.month][0:3]
+ for cve in get_edge_cves(args.year, month):
+ print(cve)
+
+ elif args.bug:
+ for bug in args.bug:
+ cves = get_cve_from_bug_alias(bug)
+
+ msrcs = []
+ for cve in cves:
+ msrcs.append(get_msrc_for_cve(cve))
+
+ # Dedupe
+ msrcs = list(set(msrcs))
+
+ for msrc in msrcs:
+ for cve in get_edge_cves(msrc.split('-')[0], msrc.split('-')[1]):
+ if cve.cve in cves:
+ print(cve)
+
+ elif args.cve:
+ msrcs = []
+ cves = []
+ for cve_id in args.cve:
+ cves.append(cve_id)
+ msrcs.append(get_msrc_for_cve(cve_id))
+
+ # Dedupe
+ msrcs = list(set(msrcs))
+
+ for msrc in msrcs:
+ for cve in get_edge_cves(msrc.split('-')[0], msrc.split('-')[1]):
+ if cve.cve in cves:
+ print(cve)
+
+
+if __name__ == "__main__":
+ main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-09-26 7:25 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-09-26 7:25 UTC (permalink / raw
To: gentoo-commits
commit: 08570431fe03884631ed495d73027c53396ad024
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 26 07:24:35 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Thu Sep 26 07:25:19 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=08570431
get-edge-cves.py: Update bgo output
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get-edge-cves.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/get-edge-cves.py b/get-edge-cves.py
index 72e68f8..515c986 100755
--- a/get-edge-cves.py
+++ b/get-edge-cves.py
@@ -157,7 +157,7 @@ def get_cve_from_bug_alias(bugnumber: int) -> list[str]:
bzapi = bugzilla.Bugzilla(url, api_key)
bug = bzapi.getbug(bugnumber)
cves = bug.alias
- print(f'Bug: {bug} has {len(cves)} CVEs.')
+ print(f'Bug: {bug} has {len(cves)} CVEs:\n\t{', '.join(cves)}')
return cves
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-09-27 0:52 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-09-27 0:52 UTC (permalink / raw
To: gentoo-commits
commit: 333530ba3430c659c049d23a1b5b1a6ee23e2e41
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 26 23:14:30 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Fri Sep 27 00:52:11 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=333530ba
Trim trailing whitespace
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get-opera-version-mapping.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/get-opera-version-mapping.py b/get-opera-version-mapping.py
index 3e68ec9..ef60683 100755
--- a/get-opera-version-mapping.py
+++ b/get-opera-version-mapping.py
@@ -5,7 +5,7 @@ from bs4 import BeautifulSoup
def get_opera_chromium_versions(base_url, start_version, end_version):
"""
- Extracts Opera and Chromium versions from the given base URL with version placeholders,
+ Extracts Opera and Chromium versions from the given base URL with version placeholders,
parsing content sections for versions from start_version to end_version (inclusive).
Args:
@@ -14,7 +14,7 @@ def get_opera_chromium_versions(base_url, start_version, end_version):
end_version: The ending version to extract information for (inclusive).
Returns:
- A dictionary mapping Opera version to Chromium version.
+ A dictionary mapping Opera version to Chromium version.
If no update is mentioned, the previous Chromium version is used.
For missing data or errors, "unknown" is used.
"""
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-09-27 0:52 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-09-27 0:52 UTC (permalink / raw
To: gentoo-commits
commit: e0adc1721f392c89c8262c4f864f6b1edf796edc
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Fri Sep 27 00:46:24 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Fri Sep 27 00:52:11 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=e0adc172
get-opera-version-mapping: major refactor
- Rework the logic to get a better result when remediating
- Also store the version mapping in a dataclass (why not).
- Use packaging.version.Version to make sorting versions trivial
- Accept positional arguments for the max and min versions.
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get-opera-version-mapping.py | 118 +++++++++++++++++++++++++++++++------------
1 file changed, 86 insertions(+), 32 deletions(-)
diff --git a/get-opera-version-mapping.py b/get-opera-version-mapping.py
index 6d6f3de..015fd21 100755
--- a/get-opera-version-mapping.py
+++ b/get-opera-version-mapping.py
@@ -1,6 +1,32 @@
#!/usr/bin/env python
+
+# SPDX-License-Identifier: GPL-2.0-or-later
+# This script is used to extract Opera and Chromium versions from the Opera changelog (blog)
+# This is incomplete data, so we need to fill in the gaps with the Chromium version from the previous known version
+# The intent here is to have _some_ sort of datasource to identify a potentially-fixed version of Opera based on
+# the Chromium version it includes.
+# High level logic:
+# We can fetch the opera blog posts that relate to a major version of Opera as long as they don't change their URIs.
+# We iterate over H4 elements to get the Opera version (and date, though we throw that away)
+# We then iterate over child elements until we find an "Update Chromium" entry, which we can use to get the
+# Chromium version (in which case we bail early) Or we exhaust the children and give up.
+# Lather, rinse, repeat.
+
+import argparse, dataclasses
+
import requests
from bs4 import BeautifulSoup
+from packaging.version import Version
+
+
+@dataclasses.dataclass
+class OperaChromiumVersion:
+ opera_version: Version
+ chromium_version: Version
+
+ def __str__(self):
+ chromium_version_str = 'unknown' if self.chromium_version == Version('0.0.0.0') else str(self.chromium_version)
+ return f"Opera Version: {self.opera_version}, Chromium Version: {chromium_version_str}"
def get_opera_chromium_versions(base_url, start_version, end_version):
@@ -15,16 +41,11 @@ def get_opera_chromium_versions(base_url, start_version, end_version):
end_version: The ending version to extract information for (inclusive).
Returns:
- A dictionary mapping Opera version to Chromium version.
- If no update is mentioned, the previous Chromium version is used.
- For missing data or errors, "unknown" is used.
+ A list of OperaChromiumVersion objects containing the extracted version information.
"""
- versions = {}
- chromium_version = None
+ versions: list[OperaChromiumVersion] = []
for version in range(start_version, end_version + 1):
- # Fix formatting issue:
- # OR url = base_url.format(version)
url = base_url.format(version)
print(f"Processing version {version}")
@@ -38,8 +59,8 @@ def get_opera_chromium_versions(base_url, start_version, end_version):
# Iterate through each section starting with an H4 element
for section in content.find_all('h4'):
+ chromium_version = None
version_str, date_str = section.text.strip().split(' – ')
- versions[version_str] = chromium_version
# Process all content elements (including nested ones) until the next H4
next_sibling = section.find_next_sibling(
@@ -63,7 +84,12 @@ def get_opera_chromium_versions(base_url, start_version, end_version):
# Handle missing Chromium version
if not chromium_version:
- chromium_version = "unknown"
+ chromium_version = '0.0.0.0'
+
+ versions.append(OperaChromiumVersion(
+ Version(version_str),
+ Version(chromium_version)
+ ))
except requests.exceptions.RequestException as e:
if e.args and e.args[0] == 404:
@@ -76,41 +102,69 @@ def get_opera_chromium_versions(base_url, start_version, end_version):
print(f"Unexpected error: {e}")
chromium_version = None # Reset chromium_version for next iteration
- return versions
+ # We're broadly sorted by major version, but within each major version we get newer entries first
+ # Sort by Opera version to get the correct order
+ sorted_versions = sorted(versions, key=lambda x: x.opera_version)
+ return sorted_versions
def remediate_unknown_versions(versions):
"""
- Remediates entries with "unknown" values in the versions dictionary by
+ Remediates entries with '0.0.0.0' values in the versions dictionary by
assuming no change from the previous known version.
Args:
- versions: A dictionary mapping Opera version to Chromium version.
+ versions: A list of OperaChromiumVersion objects containing the extracted version information.
Returns:
- The modified versions dictionary with "unknown" values replaced based on previous entries.
+ A list of OperaChromiumVersion objects with '0.0.0.0' values replaced
+ by the previous known version if available.
"""
- previous_version = None
- for version, chromium_version in versions.items():
- if chromium_version == "unknown":
- if previous_version is not None:
- # Update with previous version
- versions[version] = previous_version
+ previous_version: Version = Version('0.0.0.0')
+ fixed_versions: list[OperaChromiumVersion] = []
+
+ for mapping in versions:
+ if mapping.chromium_version == Version('0.0.0.0') and previous_version is not Version('0.0.0.0'):
+ # Update with previous version
+ fixed_versions.append(OperaChromiumVersion(mapping.opera_version, previous_version))
else:
- previous_version = chromium_version # Update known version for future references
- return versions
+ # This should be fine, we're always parsing from oldest to newest
+ if previous_version < mapping.chromium_version:
+ previous_version = mapping.chromium_version
+ fixed_versions.append(mapping)
+
+ return fixed_versions
+
+
+def parse_arguments():
+ """
+ Parses the command line arguments and returns the parsed values.
+
+ Returns:
+ The parsed command line arguments.
+ """
+ parser = argparse.ArgumentParser(description='Get Opera and Chromium versions.')
+ parser.add_argument('start_ver', type=int, help='starting version', default=110)
+ parser.add_argument('end_ver', type=int, help='ending version', default=115)
+ return parser.parse_args()
+
+
+def main():
+ args = parse_arguments()
+
+ # Base URL with version placeholder
+ base_url = "https://blogs.opera.com/desktop/changelog-for-{}/"
+ opera_chromium_versions = get_opera_chromium_versions(base_url, args.start_ver, args.end_ver)
+ fixed_versions = remediate_unknown_versions(opera_chromium_versions)
-# Example usage
-# Base URL with version placeholder
-base_url = "https://blogs.opera.com/desktop/changelog-for-{}/"
-opera_chromium_versions = get_opera_chromium_versions(base_url, 110, 115)
+ # Print the versions
+ if fixed_versions:
+ for mapping in fixed_versions:
+ print(mapping)
+ else:
+ print("Failed to extract any versions.")
-opera_chromium_versions = remediate_unknown_versions(opera_chromium_versions)
-if opera_chromium_versions:
- for opera_version, chromium_version in opera_chromium_versions.items():
- print(
- f"Opera Version: {opera_version}, Chromium Version: {chromium_version}")
-else:
- print("Failed to extract any versions.")
+if __name__ == "__main__":
+ main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-09-27 0:52 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-09-27 0:52 UTC (permalink / raw
To: gentoo-commits
commit: 9e4fee2f5fa978bfd98367e78bfc0fb87e3548f9
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Sep 26 23:19:37 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Fri Sep 27 00:52:11 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=9e4fee2f
flake8: Add config and do some trivial style changes
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
.flake8 | 4 ++++
get-edge-cves.py | 20 ++++++++++++++++++--
get-opera-version-mapping.py | 3 ++-
opera-bump | 1 +
4 files changed, 25 insertions(+), 3 deletions(-)
diff --git a/.flake8 b/.flake8
new file mode 100644
index 0000000..cb2f802
--- /dev/null
+++ b/.flake8
@@ -0,0 +1,4 @@
+[flake8]
+ignore = E401
+max-line-length = 120
+max-complexity = 15
diff --git a/get-edge-cves.py b/get-edge-cves.py
index 515c986..44b2eef 100755
--- a/get-edge-cves.py
+++ b/get-edge-cves.py
@@ -70,6 +70,18 @@ class EdgeCVE:
def get_edge_cves(year, month) -> list[EdgeCVE]:
+ """
+ Queries the Microsoft Security Response Center (MSRC) API for the Common Vulnerability Reporting Framework (CVRF)
+ for a given month and extracts the Chromium version mapping for Microsoft Edge (Chromium-based) from the CVRF.
+
+ Args:
+ year: The year to query.
+ month: The month to query.
+
+ Returns:
+ list[EdgeCVE]: A list of EdgeCVE objects.
+ """
+
msrcapi = f"https://api.msrc.microsoft.com/cvrf/v3.0/cvrf/{year}-{month}"
# Get the CVRF for the specified month
@@ -106,7 +118,7 @@ def get_edge_cves(year, month) -> list[EdgeCVE]:
# Fall back to parsing that horrible, horrible table in the notes
notes = vulnerability.find(".//{http://www.icasi.org/CVRF/schema/vuln/1.1}Notes")
# There appear to be multiple notes, but only one has content that we want:
- # <vuln:Note Title="FAQ" Type="FAQ" Ordinal="10"><p><strong>What is the version information for this release?</strong></p>
+ # <vuln:Note Title="FAQ" Type="FAQ" Ordinal="10"><p><strong>What is the version information for this release?</strong></p> # noqa: E501
found = False
for note in notes:
if note.attrib['Title'] == "FAQ" and note.attrib['Type'] == "FAQ":
@@ -121,7 +133,7 @@ def get_edge_cves(year, month) -> list[EdgeCVE]:
if len(rows) > 1:
cells = rows[1].find_all('td')
if len(cells) > 1:
- # We want the second cell (The first is the channel, the third the chromium version it's based on)
+ # We want the second cell (1st is channel, 3rd is chromium version)
edge_version = cells[1].text
if portage_versions.ververify(edge_version):
found = True
@@ -197,11 +209,14 @@ def parse_arguments():
def main():
args = parse_arguments()
+ # If we have a CVE to query (bugs contain them in the Alias field) we can query the API directly
+ # and work out which CVRF(s) to query.
if not args.bug and not args.cve:
month = calendar.month_name[args.month][0:3]
for cve in get_edge_cves(args.year, month):
print(cve)
+ # If we have a bug, we can query the bugzilla API to get the CVEs associated with it
elif args.bug:
for bug in args.bug:
cves = get_cve_from_bug_alias(bug)
@@ -218,6 +233,7 @@ def main():
if cve.cve in cves:
print(cve)
+ # If we have a CVE (or list of CVEs), we can query the API directly to identify the CVRFs to query
elif args.cve:
msrcs = []
cves = []
diff --git a/get-opera-version-mapping.py b/get-opera-version-mapping.py
index ef60683..6d6f3de 100755
--- a/get-opera-version-mapping.py
+++ b/get-opera-version-mapping.py
@@ -9,7 +9,8 @@ def get_opera_chromium_versions(base_url, start_version, end_version):
parsing content sections for versions from start_version to end_version (inclusive).
Args:
- base_url: The base URL for Opera changelogs with a version placeholder (e.g., "https://blogs.opera.com/desktop/changelog-for-{version}/").
+ base_url: The base URL for Opera changelogs with a version placeholder (e.g.,
+ "https://blogs.opera.com/desktop/changelog-for-{version}/").
start_version: The starting version to extract information for (inclusive).
end_version: The ending version to extract information for (inclusive).
diff --git a/opera-bump b/opera-bump
index c1e3c46..9f6a964 100755
--- a/opera-bump
+++ b/opera-bump
@@ -393,5 +393,6 @@ def main():
f"www-client/{pkg}: remove old",
"-s", "-S")
+
if __name__ == "__main__":
main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-10-10 10:03 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-10-10 10:03 UTC (permalink / raw
To: gentoo-commits
commit: 97423cbcc271276d24899f528c879ab011f28d1b
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Oct 10 10:01:54 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Thu Oct 10 10:01:54 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=97423cbc
New script: iterate-over-ebuild.sh
Iterates over a chromium ebuild provided as an argument and
updates `keeplibs` when the build fails.
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
iterate-over-ebuild.sh | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)
diff --git a/iterate-over-ebuild.sh b/iterate-over-ebuild.sh
new file mode 100755
index 0000000..5013a45
--- /dev/null
+++ b/iterate-over-ebuild.sh
@@ -0,0 +1,59 @@
+#!/bin/bash
+# spdx-license-identifier: GPL-2.0-or-later
+# Script to iterate over `ebuild foo-1.2.3.ebuild clean merge` and automatically add values to keeplibs.
+# Usage: ./iterate-over-ebuild.sh foo-1.2.3.ebuild
+# This script will run until the ebuild is merged, or until you interrupt it with Ctrl+C.
+# It will add the libraries to keeplibs in the ebuild as it goes.
+
+package="${1%.ebuild}"
+tmpfile=$(mktemp)
+iter=0
+added=()
+
+# Trap for Ctrl+C
+trap 'cleanup' INT
+
+cleanup() {
+ echo "[$(date)]: Script interrupted."
+ echo "$tmpfile" for this iteration\'s logs.
+ exit 1
+}
+
+while true; do
+ libs=()
+ echo "[$(date)]: Processing $package; iteration $((++iter))"
+ echo "So far, we've added:"
+ if [ ${#added[@]} -eq 0 ]; then
+ echo " Nothing"
+ fi
+ for i in "${added[@]}"; do
+ echo " $i"
+ done
+ ebuild "${1}" clean merge 2>&1 | tee "$tmpfile"
+
+ # Should only ever be one but whatever
+ mapfile -t libs < <(grep 'ninja: error:' "$tmpfile" | awk '{print $3}' | cut -c 8- | awk -F/ '{OFS="/"; NF--; print}')
+
+ if [ ${#libs[@]} -eq 0 ]; then
+ echo "[$(date)]: No new libraries to whitelist."
+ else
+ for lib in "${libs[@]}"; do
+ echo "[$(date)]: Whitelisting $lib"
+ if grep -q "$lib" "${1}"; then
+ # Something went wrong if we're here but whatever.
+ echo "[$(date)]: $lib already exists in keeplibs"
+ else
+ echo "[$(date)]: Adding $lib to keeplibs"
+ sed -i "/^\s*local keeplibs=/a \t\t$lib" "${1}"
+ added+=("$lib")
+ fi
+ done
+ fi
+
+ if grep -q "www-client/$package merged" "$tmpfile"; then
+ rm "$tmpfile"
+ break
+ fi
+ # Start with a clean slate for the next iteration
+ rm "$tmpfile"
+done
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-10-10 21:52 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-10-10 21:52 UTC (permalink / raw
To: gentoo-commits
commit: 563207f4a244e2ca02115224760a9db16eeacbcf
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Thu Oct 10 21:51:32 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Thu Oct 10 21:52:05 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=563207f4
Make sure we don't match substrings and loop forever
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
iterate-over-ebuild.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/iterate-over-ebuild.sh b/iterate-over-ebuild.sh
index 5013a45..9d0d479 100755
--- a/iterate-over-ebuild.sh
+++ b/iterate-over-ebuild.sh
@@ -39,12 +39,12 @@ while true; do
else
for lib in "${libs[@]}"; do
echo "[$(date)]: Whitelisting $lib"
- if grep -q "$lib" "${1}"; then
+ if grep -q "$lib$" "${1}"; then
# Something went wrong if we're here but whatever.
echo "[$(date)]: $lib already exists in keeplibs"
else
echo "[$(date)]: Adding $lib to keeplibs"
- sed -i "/^\s*local keeplibs=/a \t\t$lib" "${1}"
+ sed -i "/^\s*local keeplibs=/a $lib" "${1}"
added+=("$lib")
fi
done
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-10-23 3:50 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-10-23 3:50 UTC (permalink / raw
To: gentoo-commits
commit: 8325458dc3fc907432f50d0710909cba9144834e
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Wed Oct 23 03:47:39 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Wed Oct 23 03:48:04 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=8325458d
Fail after a (configurable) timeout
If we make it more than say 5 minutes we've probably gotten past
the point where GN would complain and any failures are likely to
require actual dev effort.
Just die with an error at that point to save on electricity.
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
iterate-over-ebuild.sh | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/iterate-over-ebuild.sh b/iterate-over-ebuild.sh
index 9d0d479..7cd0f64 100755
--- a/iterate-over-ebuild.sh
+++ b/iterate-over-ebuild.sh
@@ -9,6 +9,7 @@ package="${1%.ebuild}"
tmpfile=$(mktemp)
iter=0
added=()
+timeout_secs=300
# Trap for Ctrl+C
trap 'cleanup' INT
@@ -20,6 +21,7 @@ cleanup() {
}
while true; do
+ start_time=$(date +%s)
libs=()
echo "[$(date)]: Processing $package; iteration $((++iter))"
echo "So far, we've added:"
@@ -54,6 +56,15 @@ while true; do
rm "$tmpfile"
break
fi
+
+ end_time=$(date +%s)
+ elapsed_time=$((end_time - start_time))
+ if [ $elapsed_time -gt $timeout_secs ]; then
+ echo "[$(date)]: Ebuild execution took longer than the timeout. This is likely a build failure that requires patching. Exiting."
+ echo "$tmpfile" for this iteration\'s logs.
+ exit 1
+ fi
+
# Start with a clean slate for the next iteration
rm "$tmpfile"
done
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [gentoo-commits] proj/chromium-tools:master commit in: /
@ 2024-10-23 3:50 Matt Jolly
0 siblings, 0 replies; 39+ messages in thread
From: Matt Jolly @ 2024-10-23 3:50 UTC (permalink / raw
To: gentoo-commits
commit: d19c0cfce1aaee8399e553317e5541809cc5354b
Author: Matt Jolly <kangie <AT> gentoo <DOT> org>
AuthorDate: Wed Oct 23 03:46:36 2024 +0000
Commit: Matt Jolly <kangie <AT> gentoo <DOT> org>
CommitDate: Wed Oct 23 03:49:42 2024 +0000
URL: https://gitweb.gentoo.org/proj/chromium-tools.git/commit/?id=d19c0cfc
extract test_fonts sha
Signed-off-by: Matt Jolly <kangie <AT> gentoo.org>
get-chromium-toolchain-strings.py | 154 +++++++++++++++++++++++++++++++++++++-
1 file changed, 151 insertions(+), 3 deletions(-)
diff --git a/get-chromium-toolchain-strings.py b/get-chromium-toolchain-strings.py
index 0582912..d7c124d 100755
--- a/get-chromium-toolchain-strings.py
+++ b/get-chromium-toolchain-strings.py
@@ -3,12 +3,154 @@
# This script extracts the revision and sub-revision from the update.py and update_rust.py files in the Chromium source code.
# The revision and sub-revision are used to identify the version of Clang and Rust used in the Chromium toolchain.
-
+import json
import requests
import sys
+def get_testfonts(url) -> str:
+ """
+ Reads the DEPS file (gclient) and extracts the testfonts SHA which is used
+ as the object name (SHA256 as of 2024)
+ deps['src/third_party/test_fonts']['objects'][0]['object_name']
+
+ Args:
+ url (str): The URL of the DEPS file on GitHub's raw endpoint.
+
+ Returns:
+ str: The SHA256 of the testfonts, or None if not found.
+ """
+
+ # We're not properly parsing the DEPS file, but it's 'close enough' to JSON that
+ # we can throw away the preamble and do some remediation to read the values in.
+
+ testfonts = None
+ response = requests.get(url)
+ if response.status_code == 200:
+ text = response.content.decode('utf-8')
+ lines = text.splitlines()
+ # throw away everything up to `deps = {`
+ # We'll add our own opening brace to make it valid JSON
+ start = 0
+ for idx, line in enumerate(lines):
+ if line.startswith("deps = {"):
+ start = idx + 1
+ break
+
+ # throw away everything after the variable ends `}`
+ length = 0
+ for idx, line in enumerate(lines):
+ if idx < start:
+ continue
+ if line.startswith("}"):
+ length = idx
+ break
+
+ deps: list[str] = ['{', '}']
+ deps[1:1] = lines[start:length]
+
+ # remove any comments, because JSON doesn't like them
+ deps = [line for line in deps if not line.strip().startswith('#')]
+
+ # I hate to do this, but we need to remediate the JSON - single quotes to double quotes ho!
+ deps = [line.replace("'", '"') for line in deps]
+ # the `condition` variable is always a python comparison. Let's not even try to parse it.
+ # we don't care so just drop the whole line
+ deps = [line for line in deps if "condition" not in line]
+ # ditto `Var()`
+ deps = [line for line in deps if "Var(" not in line]
+ # if a line ends in ' +' it's a python thing and we probably already stripped whatever is being
+ # concatenated, so we can just remove the '+' and append a ','.
+ deps = [line.replace(" +", ",") if line.endswith(" +") else line for line in deps]
+ # strip ' "@",' from any lines... No idea what gclient does with this
+ deps = [line.replace(' "@",', "") for line in deps]
+
+
+ # If we encounter '[{' or '}]' we should expand them onto individual lines.
+ # for '[{', remove the { and add it on a new line, for '}]' remove the ] and add it on a new line.
+ # every instance so far has been '}],' so let's assume that holds true?
+ newdeps = []
+ for line in deps:
+ if '[{' in line:
+ # 'blah: [', '{'
+ newdeps.append(line[:-1])
+ newdeps.append('{')
+ elif '}]' in line:
+ # '},', '],'
+ newdeps.append(line[:-2])
+ newdeps.append('],')
+ else:
+ newdeps.append(line)
-def get_revision_info(url):
+ deps = newdeps
+
+ # if the last thing in an object has a trailing comma, it's invalid JSON so we need to remove it,
+ # probably easiest to do if we check that the next line is '}' when stripped and remediate that
+ newdeps = []
+ for idx, line in enumerate(deps):
+ if line.endswith(",") and deps[idx + 1].strip() == "}":
+ newdeps.append(line.replace(",", ""))
+ elif line.endswith(",") and deps[idx + 1].strip() == "},":
+ newdeps.append(line.replace(",", ""))
+ else:
+ newdeps.append(line)
+
+ deps = newdeps
+ newdeps = []
+
+ for idx, line in enumerate(deps):
+ if line.endswith("},") and deps[idx + 1].strip() == "]":
+ newdeps.append(line.replace(",", ""))
+ elif line.endswith("},") and deps[idx + 1].strip() == "],":
+ newdeps.append(line.replace(",", ""))
+ else:
+ newdeps.append(line)
+
+ deps = newdeps
+
+ # If the line does not contain a colon _and_ the previous and next lines contain '{' and '}' respectively,
+ # it's very likely a naked sha and json can't parse it. We can just strip it.
+ newdeps = []
+ for idx, line in enumerate(deps):
+ if ":" not in line and "{" in deps[idx - 1] and '}' in deps[idx + 1]:
+ continue
+ else:
+ newdeps.append(line)
+
+ deps = newdeps
+
+ # final blacklist; not worth writing a rule for this
+ bad_lines = [
+ '+ "@" + "42e892d96e47b1f6e29844cc705e148ec4856448", # release 1.9.4',
+ ]
+ deps = [line for line in deps if line.strip() not in bad_lines]
+
+ # Clean up any keys with no values. Always do this last
+ newdeps = []
+ for idx, line in enumerate(deps):
+ if line.endswith(":") and deps[idx + 1].strip() == "":
+ continue
+ else:
+ newdeps.append(line)
+
+ deps = newdeps
+
+ # debug_lines = range(1460, 1500)
+ # for idx, line in enumerate(deps):
+ # if idx in debug_lines:
+ # print(f"{idx}: {line}")
+
+ # Now we have a list of strings that should be valid JSON
+ # We can join them and load them
+ deps = json.loads('\n'.join(deps))
+ # Now we can get the testfonts SHA
+ return deps['src/third_party/test_fonts/test_fonts']['objects'][0]['object_name']
+ else:
+ raise ValueError(f"Failed to get revision info. Status code: {response.status_code}")
+
+ return testfonts
+
+
+def get_revision_info(url) -> str:
"""
Extracts revision and sub-revision from a Chromium source file URL.
@@ -16,7 +158,7 @@ def get_revision_info(url):
url (str): The URL of the source file on GitHub's raw endpoint.
Returns:
- tuple: A tuple containing the revision (str) and sub-revision (int),
+ tuple: A tuple containing the revision (str) and sub-revision (int)
or (None, None) if not found.
"""
response = requests.get(url)
@@ -48,8 +190,10 @@ def main():
base_url = "https://raw.githubusercontent.com/chromium/chromium/"
clang_url = f"{base_url}{version}/tools/clang/scripts/update.py"
rust_url = f"{base_url}{version}/tools/rust/update_rust.py"
+ deps_url = f"{base_url}{version}/DEPS"
clang_revision, clang_sub_revision = get_revision_info(clang_url)
rust_revision, rust_sub_revision = get_revision_info(rust_url)
+ testfonts = get_testfonts(deps_url)
if clang_revision and clang_sub_revision:
print(f"clang revision: {clang_revision}-{clang_sub_revision}")
else:
@@ -58,6 +202,10 @@ def main():
print(f"rust revision: {rust_revision}-{rust_sub_revision}")
else:
print("rust revision not found")
+ if testfonts:
+ print(f"test fonts: {testfonts}")
+ else:
+ print("test fonts not found")
if __name__ == "__main__":
main()
^ permalink raw reply related [flat|nested] 39+ messages in thread
end of thread, other threads:[~2024-10-23 3:50 UTC | newest]
Thread overview: 39+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-11-21 19:34 [gentoo-commits] proj/chromium-tools:master commit in: / Stephan Hartmann
-- strict thread matches above, loose matches on Subject: below --
2024-10-23 3:50 Matt Jolly
2024-10-23 3:50 Matt Jolly
2024-10-10 21:52 Matt Jolly
2024-10-10 10:03 Matt Jolly
2024-09-27 0:52 Matt Jolly
2024-09-27 0:52 Matt Jolly
2024-09-27 0:52 Matt Jolly
2024-09-26 7:25 Matt Jolly
2024-09-26 5:29 Matt Jolly
2024-09-26 5:21 Matt Jolly
2024-09-26 3:03 Matt Jolly
2024-09-26 2:39 Matt Jolly
2024-09-26 2:36 Matt Jolly
2024-08-30 3:39 Matt Jolly
2024-06-01 7:22 Matt Jolly
2024-05-31 23:02 Matt Jolly
2024-03-28 2:39 Matt Jolly
2024-03-20 21:45 Matt Jolly
2024-03-20 21:45 Matt Jolly
2024-03-20 21:45 Matt Jolly
2024-03-20 21:45 Matt Jolly
2023-02-05 15:09 Stephan Hartmann
2022-09-01 19:33 Mike Gilbert
2022-09-01 19:24 Mike Gilbert
2022-05-06 9:55 Stephan Hartmann
2022-05-03 16:54 Mike Gilbert
2022-05-03 16:54 Mike Gilbert
2022-02-11 17:16 Stephan Hartmann
2022-02-05 16:29 Stephan Hartmann
2022-01-31 20:20 Stephan Hartmann
2020-10-26 17:48 Mike Gilbert
2016-09-15 16:15 Mike Gilbert
2016-09-15 16:11 Mike Gilbert
2015-08-13 20:53 Mike Gilbert
2012-07-31 23:27 Mike Gilbert
2012-07-31 20:39 Mike Gilbert
2012-06-18 7:38 Paweł Hajdan
2011-10-25 16:36 Paweł Hajdan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox