[gentoo-commits] proj/tinderbox-cluster:master commit in: buildbot_gentoo_ci/db/

[gentoo-commits] proj/tinderbox-cluster:master commit in: buildbot_gentoo_ci/db/

[Magnus Granberg]

commit:     503420b53571da6ee3a2b7db8fa9407337a19162
Author:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
AuthorDate: Sun Jan 10 21:33:45 2021 +0000
Commit:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
CommitDate: Sun Jan 10 21:33:45 2021 +0000
URL:        https://gitweb.gentoo.org/proj/tinderbox-cluster.git/commit/?id=503420b5

Update Copyright on db.repositorys.py

Signed-off-by: Magnus Granberg <zorry <AT> gentoo.org>

 buildbot_gentoo_ci/db/repositorys.py | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/buildbot_gentoo_ci/db/repositorys.py b/buildbot_gentoo_ci/db/repositorys.py
index d93ee72..4660530 100644
--- a/buildbot_gentoo_ci/db/repositorys.py
+++ b/buildbot_gentoo_ci/db/repositorys.py
@@ -1,5 +1,21 @@
+# This file has parts from Buildbot and is modifyed by Gentoo Authors.
+# Buildbot is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, version 2.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc., 51
+# Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Copyright Buildbot Team Members
+# Origins: buildbot.db.*
+# Modifyed by Gentoo Authors.
 # Copyright 2021 Gentoo Authors
-# Distributed under the terms of the GNU General Public License v2
 
 import sqlalchemy as sa
 

[gentoo-commits] proj/tinderbox-cluster:master commit in: buildbot_gentoo_ci/db/

[Magnus Granberg]

commit:     f06629a2b072f7a93d74a402897c1baaf6010a14
Author:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
AuthorDate: Thu Mar 18 23:13:07 2021 +0000
Commit:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
CommitDate: Thu Mar 18 23:13:07 2021 +0000
URL:        https://gitweb.gentoo.org/proj/tinderbox-cluster.git/commit/?id=f06629a2

JSON format don't support datetime format in version data

Signed-off-by: Magnus Granberg <zorry <AT> gentoo.org>

 buildbot_gentoo_ci/db/versions.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/buildbot_gentoo_ci/db/versions.py b/buildbot_gentoo_ci/db/versions.py
index df9b4d3..85bd105 100644
--- a/buildbot_gentoo_ci/db/versions.py
+++ b/buildbot_gentoo_ci/db/versions.py
@@ -24,7 +24,7 @@ from twisted.internet import defer
 
 from buildbot.db import base
 from buildbot.util import epoch2datetime
-
+from buildbot.util import datetime2epoch
 class VersionsConnectorComponent(base.DBConnectorComponent):
 
     @defer.inlineCallbacks
@@ -110,5 +110,5 @@ class VersionsConnectorComponent(base.DBConnectorComponent):
             file_hash=row.file_hash,
             commit_id=row.commit_id,
             deleted=row.deleted,
-            deleted_at=row.deleted_at
+            deleted_at=datetime2epoch(row.deleted_at)
             )

[gentoo-commits] proj/tinderbox-cluster:master commit in: buildbot_gentoo_ci/db/

[Magnus Granberg]

commit:     60ce5fec404770c2dbc58352808397f7e9b87c93
Author:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
AuthorDate: Thu Jul 28 11:31:07 2022 +0000
Commit:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
CommitDate: Thu Jul 28 11:31:07 2022 +0000
URL:        https://gitweb.gentoo.org/proj/tinderbox-cluster.git/commit/?id=60ce5fec

Add support for gitlab/github project name in project db

Signed-off-by: Magnus Granberg <zorry <AT> gentoo.org>

 buildbot_gentoo_ci/db/model.py    | 2 ++
 buildbot_gentoo_ci/db/projects.py | 1 +
 2 files changed, 3 insertions(+)

diff --git a/buildbot_gentoo_ci/db/model.py b/buildbot_gentoo_ci/db/model.py
index d6e9860..be305f2 100644
--- a/buildbot_gentoo_ci/db/model.py
+++ b/buildbot_gentoo_ci/db/model.py
@@ -109,6 +109,8 @@ class Model(base.DBConnectorComponent):
                   default=lambda: str(uuid.uuid4())),
         # project's name
         sa.Column('name', sa.String(255), nullable=False),
+        # project name on gitlab/github
+        sa.Column('git_project_name', sa.String(255), nullable=False),
         # description of the project
         sa.Column('description', sa.Text, nullable=True),
         sa.Column('profile', sa.String(255), nullable=False),

diff --git a/buildbot_gentoo_ci/db/projects.py b/buildbot_gentoo_ci/db/projects.py
index 936cd9d..8e2cd6f 100644
--- a/buildbot_gentoo_ci/db/projects.py
+++ b/buildbot_gentoo_ci/db/projects.py
@@ -216,6 +216,7 @@ class ProjectsConnectorComponent(base.DBConnectorComponent):
         return dict(
             uuid=row.uuid,
             name=row.name,
+            git_project_name=row.git_project_name,
             description=row.description,
             profile=row.profile,
             profile_repository_uuid=row.profile_repository_uuid,

[gentoo-commits] proj/tinderbox-cluster:master commit in: buildbot_gentoo_ci/db/

[Magnus Granberg]

commit:     9e94e5e0e464ba589d25d60d6891c0a8dee0ecd7
Author:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
AuthorDate: Mon Aug 22 22:54:39 2022 +0000
Commit:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
CommitDate: Mon Aug 22 22:54:39 2022 +0000
URL:        https://gitweb.gentoo.org/proj/tinderbox-cluster.git/commit/?id=9e94e5e0

Add Merge Requeste test to projects_repositorys in db

Signed-off-by: Magnus Granberg <zorry <AT> gentoo.org>

 buildbot_gentoo_ci/db/model.py    | 1 +
 buildbot_gentoo_ci/db/projects.py | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/buildbot_gentoo_ci/db/model.py b/buildbot_gentoo_ci/db/model.py
index be305f2..3f1dc0a 100644
--- a/buildbot_gentoo_ci/db/model.py
+++ b/buildbot_gentoo_ci/db/model.py
@@ -144,6 +144,7 @@ class Model(base.DBConnectorComponent):
         sa.Column('pkgcheck', sa.Enum('package','full','none'), default='none'),
         sa.Column('build', sa.Boolean, default=False),
         sa.Column('test', sa.Boolean, default=False),
+        sa.Column('test_mr', sa.Boolean, default=False),
     )
 
     # projects etc/portage settings

diff --git a/buildbot_gentoo_ci/db/projects.py b/buildbot_gentoo_ci/db/projects.py
index 8e2cd6f..9dcfa45 100644
--- a/buildbot_gentoo_ci/db/projects.py
+++ b/buildbot_gentoo_ci/db/projects.py
@@ -241,7 +241,8 @@ class ProjectsConnectorComponent(base.DBConnectorComponent):
             auto=row.auto,
             pkgcheck=pkgcheck,
             build=row.build,
-            test=row.test
+            test=row.test,
+            test_mr=row.test_mr
             )
 
     def _row2dict_projects_workers(self, conn, row):

[gentoo-commits] proj/tinderbox-cluster:master commit in: buildbot_gentoo_ci/db/

[Magnus Granberg]

commit:     4827e44a0e0e6df8786bec353c7f01a39af4e089
Author:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
AuthorDate: Mon Oct  3 01:20:32 2022 +0000
Commit:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
CommitDate: Mon Oct  3 01:20:32 2022 +0000
URL:        https://gitweb.gentoo.org/proj/tinderbox-cluster.git/commit/?id=4827e44a

Fix bootstrap_tab in workers db

Signed-off-by: Magnus Granberg <zorry <AT> gentoo.org>

 buildbot_gentoo_ci/db/workers.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/buildbot_gentoo_ci/db/workers.py b/buildbot_gentoo_ci/db/workers.py
index afc7815..e82a6b2 100644
--- a/buildbot_gentoo_ci/db/workers.py
+++ b/buildbot_gentoo_ci/db/workers.py
@@ -151,7 +151,7 @@ class WorkersConnectorComponent(base.DBConnectorComponent):
             name=row.name,
             tag=row.tag,
             dockerfile=row.dockerfile,
-            bootstrap_image_tag=row.bootstrap_image_tag
+            bootstrap_tag=row.bootstrap_tag
             )
     def _row2dict_worker_config(self, conn, row):
         return dict(

[gentoo-commits] proj/tinderbox-cluster:master commit in: buildbot_gentoo_ci/db/

[Magnus Granberg]

commit:     165def4478ff2558075c7dcabdff85b2be31277b
Author:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
AuthorDate: Fri Mar 10 22:32:06 2023 +0000
Commit:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
CommitDate: Fri Mar 10 22:32:06 2023 +0000
URL:        https://gitweb.gentoo.org/proj/tinderbox-cluster.git/commit/?id=165def44

Fix 2 bugs in db for email

Signed-off-by: Magnus Granberg <zorry <AT> gentoo.org>

 buildbot_gentoo_ci/db/packages.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/buildbot_gentoo_ci/db/packages.py b/buildbot_gentoo_ci/db/packages.py
index 79ae685..1f68637 100644
--- a/buildbot_gentoo_ci/db/packages.py
+++ b/buildbot_gentoo_ci/db/packages.py
@@ -210,7 +210,8 @@ class PackagesConnectorComponent(base.DBConnectorComponent):
             tbl = self.db.model.packages_emails
             conn.execute(tbl.delete(
                 whereclause=((tbl.c.package_uuid == package_uuid))))
-        return self.db.pool.do(thd)
+        res = yield self.db.pool.do(thd)
+        return res
 
     def _row2dict_email(self, conn, row):
         return dict(
@@ -222,6 +223,6 @@ class PackagesConnectorComponent(base.DBConnectorComponent):
             id=row.id,
             email_id=row.email_id,
             package_uuid=row.package_uuid,
-            type=row.type,
+            mail_type=row.mail_type,
             proxied=row.proxied,
             )

[gentoo-commits] proj/tinderbox-cluster:master commit in: buildbot_gentoo_ci/db/

[Magnus Granberg]

commit:     2dfba1de696c99c0c03a8c1c6abd878b60f3c7fc
Author:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
AuthorDate: Sun May 28 09:48:40 2023 +0000
Commit:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
CommitDate: Sun May 28 09:48:40 2023 +0000
URL:        https://gitweb.gentoo.org/proj/tinderbox-cluster.git/commit/?id=2dfba1de

Update db python code to bb3.8

Signed-off-by: Magnus Granberg <zorry <AT> gentoo.org>

 buildbot_gentoo_ci/db/connector.py |  35 +++++-
 buildbot_gentoo_ci/db/model.py     | 231 ++++++++++++++++++-------------------
 2 files changed, 141 insertions(+), 125 deletions(-)

diff --git a/buildbot_gentoo_ci/db/connector.py b/buildbot_gentoo_ci/db/connector.py
index 7665f84..0cc7884 100644
--- a/buildbot_gentoo_ci/db/connector.py
+++ b/buildbot_gentoo_ci/db/connector.py
@@ -15,7 +15,7 @@
 # Copyright Buildbot Team Members
 # Origins: buildbot.db.connector.py
 # Modifyed by Gentoo Authors.
-# Copyright 2021 Gentoo Authors
+# Copyright 2023 Gentoo Authors
 
 import textwrap
 
@@ -70,6 +70,10 @@ class DBConnector(service.ReconfigurableServiceMixin,
         self.setName('db')
         self.basedir = basedir
 
+        # not configured yet - we don't build an engine until the first
+        # reconfig
+        self.configured_url = None
+
         # set up components
         self._engine = None  # set up in reconfigService
         self.pool = None  # set up in reconfigService
@@ -88,12 +92,16 @@ class DBConnector(service.ReconfigurableServiceMixin,
         self.builds = builds.BuildsConnectorComponent(self)
         self.workers = workers.WorkersConnectorComponent(self)
 
+        self.cleanup_timer = internet.TimerService(self.CLEANUP_PERIOD,
+                                                   self._doCleanup)
+        self.cleanup_timer.clock = self.master.reactor
+        yield self.cleanup_timer.setServiceParent(self)
+
     @defer.inlineCallbacks
     def setup(self, config, check_version=True, verbose=True):
-        db_url = config.db['db_url']
+        db_url = self.configured_url = config.db['db_url']
 
-        log.msg("Setting up database with URL %r"
-                % util.stripUrlPassword(db_url))
+        log.msg(f"Setting up database with URL {repr(util.stripUrlPassword(db_url))}")
 
         # set up the engine and pool
         self._engine = enginestrategy.create_engine(db_url,
@@ -113,3 +121,22 @@ class DBConnector(service.ReconfigurableServiceMixin,
                 for l in upgrade_message.format(basedir=self.basedir).split('\n'):
                     log.msg(l)
                 raise exceptions.DatabaseNotReadyError()
+
+    def reconfigServiceWithBuildbotConfig(self, new_config):
+        # double-check -- the master ensures this in config checks
+        assert self.configured_url == new_config.db['db_url']
+
+        return super().reconfigServiceWithBuildbotConfig(new_config)
+
+    def _doCleanup(self):
+        """
+        Perform any periodic database cleanup tasks.
+        @returns: Deferred
+        """
+        # pass on this if we're not configured yet
+        if not self.configured_url:
+            return None
+
+        d = self.changes.pruneChanges(self.master.config.changeHorizon)
+        d.addErrback(log.err, 'while pruning changes')
+        return d

diff --git a/buildbot_gentoo_ci/db/model.py b/buildbot_gentoo_ci/db/model.py
index 7ffe0ca..b80281e 100644
--- a/buildbot_gentoo_ci/db/model.py
+++ b/buildbot_gentoo_ci/db/model.py
@@ -1,7 +1,7 @@
 # This file has parts from Buildbot and is modifyed by Gentoo Authors. 
-# Buildbot is free software: you can redistribute it and/or modify it 
-# under the terms of the GNU General Public License as published by the 
-# Free Software Foundation, version 2.
+# Buildbot is free software: you can
+# redistribute it and/or modify it under the terms of the GNU General Public
+# License as published by the Free Software Foundation, version 2.
 #
 # This program is distributed in the hope that it will be useful, but WITHOUT
 # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
@@ -18,10 +18,10 @@
 # Copyright 2023 Gentoo Authors
 
 import uuid
-import migrate
-import migrate.versioning.repository
+
+import alembic
+import alembic.config
 import sqlalchemy as sa
-from migrate import exceptions  # pylint: disable=ungrouped-imports
 
 from twisted.internet import defer
 from twisted.python import log
@@ -29,15 +29,36 @@ from twisted.python import util
 
 from buildbot.db import base
 from buildbot.db.migrate_utils import test_unicode
+from buildbot.db.types.json import JsonObject
 from buildbot.util import sautils
 
-try:
-    from migrate.versioning.schema import ControlledSchema  # pylint: disable=ungrouped-imports
-except ImportError:
-    ControlledSchema = None
+
+class UpgradeFromBefore0p9Error(Exception):
+
+    def __init__(self):
+        message = """You are trying to upgrade a buildbot 0.8.x master to buildbot 0.9.x or newer.
+        This is not supported. Please start from a clean database
+        http://docs.buildbot.net/latest/manual/upgrading/0.9-upgrade.html"""
+        # Call the base class constructor with the parameters it needs
+        super().__init__(message)
+
+
+class UpgradeFromBefore3p0Error(Exception):
+
+    def __init__(self):
+        message = """You are trying to upgrade to Buildbot 3.0 or newer from Buildbot 2.x or older.
+        This is only supported via an intermediate upgrade to newest Buildbot 2.10.x that is
+        available. Please first upgrade to 2.10.x and then try to upgrade to this version.
+        http://docs.buildbot.net/latest/manual/upgrading/3.0-upgrade.html"""
+        super().__init__(message)
 
 
 class Model(base.DBConnectorComponent):
+
+    property_name_length = 256
+    property_source_length = 256
+    hash_length = 40
+
     #
     # schema
     #
@@ -54,9 +75,6 @@ class Model(base.DBConnectorComponent):
     #
     # * dates are stored as unix timestamps (UTC-ish epoch time)
     #
-    # * sqlalchemy does not handle sa.Boolean very well on MySQL or Postgres;
-    #   use sa.SmallInteger instead
-
     # Tables related to gentoo-ci-cloud
     # -------------------------
 
@@ -452,35 +470,51 @@ class Model(base.DBConnectorComponent):
     # Migration support
     # -----------------
 
-    # this is a bit more complicated than might be expected because the first
-    # seven database versions were once implemented using a homespun migration
-    # system, and we need to support upgrading masters from that system.  The
-    # old system used a 'version' table, where SQLAlchemy-Migrate uses
-    # 'migrate_version'
+    # Buildbot has historically used 3 database migration systems:
+    #  - homegrown system that used "version" table to track versions
+    #  - SQLAlchemy-migrate that used "migrate_version" table to track versions
+    #  - alembic that uses "alembic_version" table to track versions (current)
+    # We need to detect each case and tell the user how to upgrade.
+
+    config_path = util.sibpath(__file__, "migrations/alembic.ini")
+
+    def table_exists(self, conn, table):
+        try:
+            r = conn.execute(f"select * from {table} limit 1")
+            r.close()
+            return True
+        except Exception:
+            return False
+
+    def migrate_get_version(self, conn):
+        r = conn.execute("select version from migrate_version limit 1")
+        version = r.scalar()
+        r.close()
+        return version
 
-    repo_path = util.sibpath(__file__, "migrate")
+    def alembic_get_scripts(self):
+        alembic_config = alembic.config.Config(self.config_path)
+        return alembic.script.ScriptDirectory.from_config(alembic_config)
+
+    def alembic_stamp(self, conn, alembic_scripts, revision):
+        context = alembic.runtime.migration.MigrationContext.configure(conn)
+        context.stamp(alembic_scripts, revision)
 
     @defer.inlineCallbacks
     def is_current(self):
-        if ControlledSchema is None:
-            # this should have been caught earlier by enginestrategy.py with a
-            # nicer error message
-            raise ImportError("SQLAlchemy/SQLAlchemy-Migrate version conflict")
-
-        def thd(engine):
-            # we don't even have to look at the old version table - if there's
-            # no migrate_version, then we're not up to date.
-            repo = migrate.versioning.repository.Repository(self.repo_path)
-            repo_version = repo.latest
-            try:
-                # migrate.api doesn't let us hand in an engine
-                schema = ControlledSchema(engine, self.repo_path)
-                db_version = schema.version
-            except exceptions.DatabaseNotControlledError:
+        def thd(conn):
+            if not self.table_exists(conn, 'alembic_version'):
                 return False
 
-            return db_version == repo_version
-        ret = yield self.db.pool.do_with_engine(thd)
+            alembic_scripts = self.alembic_get_scripts()
+            current_script_rev_head = alembic_scripts.get_current_head()
+
+            context = alembic.runtime.migration.MigrationContext.configure(conn)
+            current_rev = context.get_current_revision()
+
+            return current_rev == current_script_rev_head
+
+        ret = yield self.db.pool.do(thd)
         return ret
 
     # returns a Deferred that returns None
@@ -493,94 +527,49 @@ class Model(base.DBConnectorComponent):
     @defer.inlineCallbacks
     def upgrade(self):
 
-        # here, things are a little tricky.  If we have a 'version' table, then
-        # we need to version_control the database with the proper version
-        # number, drop 'version', and then upgrade.  If we have no 'version'
-        # table and no 'migrate_version' table, then we need to version_control
-        # the database.  Otherwise, we just need to upgrade it.
-
-        def table_exists(engine, tbl):
-            try:
-                r = engine.execute("select * from {} limit 1".format(tbl))
-                r.close()
-                return True
-            except Exception:
-                return False
+        # the upgrade process must run in a db thread
+        def thd(conn):
+            alembic_scripts = self.alembic_get_scripts()
+            current_script_rev_head = alembic_scripts.get_current_head()
 
-        # http://code.google.com/p/sqlalchemy-migrate/issues/detail?id=100
-        # means  we cannot use the migrate.versioning.api module.  So these
-        # methods perform similar wrapping functions to what is done by the API
-        # functions, but without disposing of the engine.
-        def upgrade(engine):
-            schema = ControlledSchema(engine, self.repo_path)
-            changeset = schema.changeset(None)
-            with sautils.withoutSqliteForeignKeys(engine):
-                for version, change in changeset:
-                    log.msg('migrating schema version {} -> {}'.format(version, version + 1))
-                    schema.runchange(version, change, 1)
-
-        def check_sqlalchemy_migrate_version():
-            # sqlalchemy-migrate started including a version number in 0.7; we
-            # support back to 0.6.1, but not 0.6.  We'll use some discovered
-            # differences between 0.6.1 and 0.6 to get that resolution.
-            version = getattr(migrate, '__version__', 'old')
-            if version == 'old':
-                try:
-                    from migrate.versioning import schemadiff
-                    if hasattr(schemadiff, 'ColDiff'):
-                        version = "0.6.1"
-                    else:
-                        version = "0.6"
-                except Exception:
-                    version = "0.0"
-            version_tup = tuple(map(int, version.split('-', 1)[0].split('.')))
-            log.msg("using SQLAlchemy-Migrate version {}".format(version))
-            if version_tup < (0, 6, 1):
-                raise RuntimeError(("You are using SQLAlchemy-Migrate {}. "
-                                    "The minimum version is 0.6.1.").format(version))
-
-        def version_control(engine, version=None):
-            ControlledSchema.create(engine, self.repo_path, version)
+            #if self.table_exists(conn, 'version'):
+            #    raise UpgradeFromBefore0p9Error()
 
-        # the upgrade process must run in a db thread
-        def thd(engine):
-            # if the migrate_version table exists, we can just let migrate
-            # take care of this process.
-            if table_exists(engine, 'migrate_version'):
-                r = engine.execute(
-                    "select version from migrate_version limit 1")
-                old_version = r.scalar()
-                if old_version < 40:
-                    raise EightUpgradeError()
-                try:
-                    upgrade(engine)
-                except sa.exc.NoSuchTableError as e:  # pragma: no cover
-                    if 'migration_tmp' in str(e):
-                        log.err('A serious error has been encountered during the upgrade. The '
-                                'previous upgrade has been likely interrupted. The database has '
-                                'been damaged and automatic recovery is impossible.')
-                        log.err('If you believe this is an error, please submit a bug to the '
-                                'Buildbot project.')
-                    raise
-
-            # if the version table exists, then we can version_control things
-            # at that version, drop the version table, and let migrate take
-            # care of the rest.
-            elif table_exists(engine, 'version'):
-                raise EightUpgradeError()
-
-            # otherwise, this db is new, so we don't bother using the migration engine
-            # and just create the tables, and put the version directly to
-            # latest
-            else:
-                # do some tests before getting started
-                test_unicode(engine)
+            if self.table_exists(conn, 'migrate_version'):
+                version = self.migrate_get_version(conn)
+
+                #if version < 40:
+                #    raise UpgradeFromBefore0p9Error()
 
+                last_sqlalchemy_migrate_version = 0
+                if version != last_sqlalchemy_migrate_version:
+                    raise UpgradeFromBefore3p0Error()
+
+                self.alembic_stamp(conn, alembic_scripts, alembic_scripts.get_base())
+                conn.execute('drop table migrate_version')
+
+            if not self.table_exists(conn, 'alembic_version'):
                 log.msg("Initializing empty database")
-                Model.metadata.create_all(engine)
-                repo = migrate.versioning.repository.Repository(self.repo_path)
 
-                version_control(engine, repo.latest)
+                # Do some tests first
+                test_unicode(conn)
+
+                Model.metadata.create_all(conn)
+                self.alembic_stamp(conn, alembic_scripts, current_script_rev_head)
+                return
+
+            context = alembic.runtime.migration.MigrationContext.configure(conn)
+            current_rev = context.get_current_revision()
+
+            if current_rev == current_script_rev_head:
+                log.msg('Upgrading database: the current database schema is already the newest')
+                return
+
+            log.msg('Upgrading database')
+            with sautils.withoutSqliteForeignKeys(conn):
+                with context.begin_transaction():
+                    context.run_migrations()
+
+            log.msg('Upgrading database: done')
 
-        check_sqlalchemy_migrate_version()
-        yield self.db.pool.do_with_engine(thd)
+        yield self.db.pool.do(thd)

[gentoo-commits] proj/tinderbox-cluster:master commit in: buildbot_gentoo_ci/db/

[Magnus Granberg]

commit:     73e8b3e7c00b1b2686ae0df6d406181b6c3808a7
Author:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
AuthorDate: Mon Jun 26 21:49:07 2023 +0000
Commit:     Magnus Granberg <zorry <AT> gentoo <DOT> org>
CommitDate: Mon Jun 26 21:49:07 2023 +0000
URL:        https://gitweb.gentoo.org/proj/tinderbox-cluster.git/commit/?id=73e8b3e7

Add support to remove ebuilds in db

Signed-off-by: Magnus Granberg <zorry <AT> gentoo.org>

 buildbot_gentoo_ci/db/versions.py | 40 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/buildbot_gentoo_ci/db/versions.py b/buildbot_gentoo_ci/db/versions.py
index 6b534f3..0aaac1b 100644
--- a/buildbot_gentoo_ci/db/versions.py
+++ b/buildbot_gentoo_ci/db/versions.py
@@ -131,6 +131,46 @@ class VersionsConnectorComponent(base.DBConnectorComponent):
         res = yield self.db.pool.do(thd)
         return res
 
+    @defer.inlineCallbacks
+    def getEbuildsByPackage(self, p_uuid, deleted=False):
+        def thd(conn):
+            tbl = self.db.model.versions
+            q = tbl.select()
+            q = q.where(tbl.c.deleted == deleted)
+            q = q.where(tbl.c.package_uuid == p_uuid)
+            return [self._row2dict(conn, row)
+                for row in conn.execute(q).fetchall()]
+        res = yield self.db.pool.do(thd)
+        return res
+
+    @defer.inlineCallbacks
+    def removeVersion(self, uuid):
+        def thd(conn, no_recurse=False):
+                tbl = self.db.model.versions
+                q = tbl.delete()
+                q = q.where(tbl.c.uuid == uuid)
+                conn.execute(q)
+        yield self.db.pool.do(thd)
+
+    @defer.inlineCallbacks
+    def removeVersionMetadata(self, version_uuid):
+        def thd(conn, no_recurse=False):
+                tbl = self.db.model.versions_metadata
+                q = tbl.delete()
+                q = q.where(tbl.c.version_uuid == version_uuid)
+                conn.execute(q)
+        yield self.db.pool.do(thd)
+
+    @defer.inlineCallbacks
+    def removeVersionKeyword(self, version_uuid):
+        def thd(conn, no_recurse=False):
+                tbl = self.db.model.versions_keywords
+                q = tbl.delete()
+                q = q.where(tbl.c.version_uuid == version_uuid)
+                conn.execute(q)
+        yield self.db.pool.do(thd)
+
+
     def _row2dict(self, conn, row):
         return dict(
             uuid=row.uuid,