[gentoo-commits] proj/R_overlay:master commit in: roverlay/remote/

public inbox for gentoo-commits@lists.gentoo.org
 help / color / mirror / Atom feed

From: "André Erdmann" <dywi@mailerd.de>
To: gentoo-commits@lists.gentoo.org
Subject: [gentoo-commits] proj/R_overlay:master commit in: roverlay/remote/
Date: Fri,  6 Jul 2012 08:15:47 +0000 (UTC)	[thread overview]
Message-ID: <1341562454.cf8a0d0dd57f04d3b9da37f091a2f401c16e5b3c.dywi@gentoo> (raw)

commit:     cf8a0d0dd57f04d3b9da37f091a2f401c16e5b3c
Author:     André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Jul  6 08:14:14 2012 +0000
Commit:     André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Jul  6 08:14:14 2012 +0000
URL:        http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=cf8a0d0d

apply distroot changes to RemoteRepo/RsyncRepo

* also added the possibility to retry rsync transfers on certain error codes

	modified:   roverlay/remote/basicrepo.py
	modified:   roverlay/remote/rsync.py

---
 roverlay/remote/basicrepo.py |    6 +++-
 roverlay/remote/rsync.py     |   56 ++++++++++++++++++++++++++++++++---------
 2 files changed, 47 insertions(+), 15 deletions(-)

diff --git a/roverlay/remote/basicrepo.py b/roverlay/remote/basicrepo.py
index 0248413..3dd09de 100644
--- a/roverlay/remote/basicrepo.py
+++ b/roverlay/remote/basicrepo.py
@@ -251,7 +251,7 @@ class RemoteRepo ( LocalRepo ):
 	"""A template for remote repositories."""
 
 	def __init__ (
-		self, name, sync_proto,
+		self, name, distroot, sync_proto,
 		directory=None,
 		src_uri=None, remote_uri=None, base_uri=None
 	):
@@ -274,7 +274,9 @@ class RemoteRepo ( LocalRepo ):
 		* | { x : x in union(src,remote,base) and x not None } | >= 1
 		 ^= at least one out of src/remote/base uri is not None
 		"""
-		super ( RemoteRepo, self ) . __init__ ( name, directory, src_uri='' )
+		super ( RemoteRepo, self ) . __init__ (
+			name, distroot, directory, src_uri=''
+		)
 
 		self.sync_proto = sync_proto
 

diff --git a/roverlay/remote/rsync.py b/roverlay/remote/rsync.py
index 4744118..9fcc348 100644
--- a/roverlay/remote/rsync.py
+++ b/roverlay/remote/rsync.py
@@ -15,6 +15,15 @@ RSYNC_ENV = util.keepenv (
 	'RSYNC_PASSWORD',
 )
 
+MAX_RSYNC_RETRY = 3
+
+RSYNC_SIGINT = 20
+
+RETRY_ON_RETCODE = frozenset ((
+	23, # "Partial transfer due to error"
+	24, # "Partial transfer due to vanished source files"
+))
+
 # TODO:
 # either reraise an KeyboardInterrupt while running rsync (which stops script
 # execution unless the interrupt is catched elsewhere) or just set a
@@ -40,7 +49,7 @@ DEFAULT_RSYNC_OPTS =  (
 class RsyncRepo ( RemoteRepo ):
 
 	def __init__ (
-		self, name,
+		self, name, distroot,
 		directory=None, src_uri=None, rsync_uri=None, base_uri=None,
 		recursive=False, extra_opts=None
 	):
@@ -59,7 +68,7 @@ class RsyncRepo ( RemoteRepo ):
 		#  using '' as remote protocol which leaves uris unchanged when
 		#   normalizing them for rsync usage
 		super ( RsyncRepo, self ) . __init__ (
-			name, '', directory=directory,
+			name, distroot=distroot, sync_proto='', directory=directory,
 			src_uri=src_uri, remote_uri=rsync_uri, base_uri=base_uri
 		)
 
@@ -103,6 +112,15 @@ class RsyncRepo ( RemoteRepo ):
 		All exceptions(?) are catched and interpreted as sync failure.
 		"""
 
+		def waitfor ( p ):
+			if p.communicate() != ( None, None ):
+				raise AssertionError ( "expected None,None from communicate!" )
+			if p.returncode == RSYNC_SIGINT:
+				raise KeyboardInterrupt ( "propagated from rsync" )
+
+			return p.returncode
+		# --- end of waitfor (...) ---
+
 		retcode = '<undef>'
 
 		try:
@@ -113,31 +131,43 @@ class RsyncRepo ( RemoteRepo ):
 
 			self.logger.debug ( 'running rsync cmd: ' + ' '.join ( rsync_cmd ) )
 
+			retry_count = 0
 
-			proc = subprocess.Popen (
-				rsync_cmd,
-				stdin=None, stdout=None, stderr=None,
-				env=RSYNC_ENV
-			)
+			proc = subprocess.Popen ( rsync_cmd, env=RSYNC_ENV )
+			retcode = waitfor ( proc )
+			del proc
 
-			if proc.communicate() != ( None, None ):
-				raise AssertionError ( "expected None,None from communicate!" )
+			while retcode in RETRY_ON_RETCODE and retry_count < MAX_RSYNC_RETRY:
+				# this handles retcodes like
+				#  * 24: "Partial transfer due to vanished source files"
+
+				# FIXME replace loop condition "retcode != 0"
+				retry_count += 1
+
+				self.logger.warning (
+					"rsync returned {!r}, retrying ((}/{})".format (
+						retcode, retry_count, MAX_RSYNC_RETRY
+					)
+				)
+
+				proc = subprocess.Popen ( rsync_cmd, env=RSYNC_ENV )
+				retcode = waitfor ( proc )
+				del proc
 
-			if proc.returncode == 0:
+			if retcode == 0:
 				self._set_ready ( is_synced=True )
 				return True
 
-			retcode = proc.returncode
 
 		except KeyboardInterrupt:
 			sys.stderr.write (
 				"\nKeyboard interrupt - waiting for rsync to exit...\n"
 			)
-			if 'proc' in locals():
+			if 'proc' in locals() and proc is not None:
 				proc.communicate()
 				retcode = proc.returncode
 			else:
-				retcode = 130
+				retcode = RSYNC_SIGINT
 
 			if RERAISE_INTERRUPT:
 				raise

next             reply	other threads:[~2012-07-06  8:16 UTC|newest]

Thread overview: 36+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-07-06  8:15 André Erdmann [this message]
  -- strict thread matches above, loose matches on Subject: below --
2025-08-09 14:24 [gentoo-commits] proj/R_overlay:master commit in: roverlay/remote/ Benda XU
2015-01-26 17:41 André Erdmann
2014-02-16 16:45 André Erdmann
2014-02-15 19:49 André Erdmann
2014-02-15 19:49 André Erdmann
2014-02-15 19:49 André Erdmann
2013-09-03 13:15 André Erdmann
2013-09-03  8:35 André Erdmann
2013-09-02 16:21 André Erdmann
2013-08-29 15:08 André Erdmann
2013-08-07 16:10 André Erdmann
2013-08-07 16:10 André Erdmann
2013-07-24  9:54 André Erdmann
2013-07-24  9:54 André Erdmann
2013-07-24  9:54 André Erdmann
2013-07-23 14:57 André Erdmann
2013-07-23  9:38 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-07-23 14:57 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-07-23  9:38 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-07-23 14:57 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-07-16 16:35 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-07-16 16:36 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2012-08-13 18:07 André Erdmann
2012-08-11  0:01 André Erdmann
2012-08-10 15:16 André Erdmann
2012-08-10 15:16 André Erdmann
2012-08-09  9:26 André Erdmann
2012-08-02 15:14 André Erdmann
2012-08-01  7:33 André Erdmann
2012-07-31 17:51 André Erdmann
2012-07-09 17:25 André Erdmann
2012-07-04 18:21 André Erdmann
2012-07-04 18:21 André Erdmann
2012-07-03 17:48 André Erdmann
2012-06-27 14:46 André Erdmann
2012-06-26 15:55 André Erdmann
2012-06-26 15:42 André Erdmann
2012-06-25 18:19 André Erdmann

find likely ancestor, descendant, or conflicting patches for this message:
( dfblob:0248413 dfblob:3dd09de dfblob:4744118 dfblob:9fcc348 )
 OR (
bs:"[gentoo-commits] proj/R_overlay:master commit in: roverlay/remote/" )
	(help)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1341562454.cf8a0d0dd57f04d3b9da37f091a2f401c16e5b3c.dywi@gentoo \
    --to=dywi@mailerd.de \
    --cc=gentoo-commits@lists.gentoo.org \
    --cc=gentoo-dev@lists.gentoo.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Be sure your reply has a Subject: header at the top and a blank line before the message body.

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox