public inbox for gentoo-portage-dev@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-portage-dev] [PATCH] EventLoop: add run_until_complete method (bug 591760)
@ 2016-08-20 22:18 Zac Medico
  2016-08-22 10:18 ` Alexander Berntsen
  2016-08-22 16:34 ` [gentoo-portage-dev] [PATCH v2] " Zac Medico
  0 siblings, 2 replies; 6+ messages in thread
From: Zac Medico @ 2016-08-20 22:18 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Zac Medico

This emulates the asyncio.AbstractEventLoop.run_until_complete(future)
interface, which will make it possible to reduce latency in situations
where it is desirable for a loop to exit at the earliest opportunity.

X-Gentoo-bug: 591760
X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760
---
 pym/portage/tests/ebuild/test_ipc_daemon.py | 23 ++++++++++++++---------
 pym/portage/util/_eventloop/EventLoop.py    | 17 ++++++++++++++++-
 2 files changed, 30 insertions(+), 10 deletions(-)

diff --git a/pym/portage/tests/ebuild/test_ipc_daemon.py b/pym/portage/tests/ebuild/test_ipc_daemon.py
index 835f51f..68f139a 100644
--- a/pym/portage/tests/ebuild/test_ipc_daemon.py
+++ b/pym/portage/tests/ebuild/test_ipc_daemon.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2015 Gentoo Foundation
+# Copyright 2010-2016 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 import tempfile
@@ -16,6 +16,7 @@ from portage.util import ensure_dirs
 from portage.util._async.ForkProcess import ForkProcess
 from portage.util._async.TaskScheduler import TaskScheduler
 from portage.util._eventloop.global_event_loop import global_event_loop
+from portage.util.futures.futures import Future
 from _emerge.SpawnProcess import SpawnProcess
 from _emerge.EbuildBuildDir import EbuildBuildDir
 from _emerge.EbuildIpcDaemon import EbuildIpcDaemon
@@ -140,19 +141,23 @@ class IpcDaemonTestCase(TestCase):
 				build_dir.unlock()
 			shutil.rmtree(tmpdir)
 
-	def _timeout_callback(self):
-		self._timed_out = True
+	def _timeout_callback(self, task_scheduler):
+		task_scheduler.cancel()
+		self._exit_callback(task_scheduler)
+
+	def _exit_callback(self, task_scheduler):
+		if not self._run_done.done():
+			self._run_done.set_result(True)
 
 	def _run(self, event_loop, task_scheduler, timeout):
-		self._timed_out = False
-		timeout_id = event_loop.timeout_add(timeout, self._timeout_callback)
+		self._run_done = Future()
+		timeout_id = event_loop.timeout_add(timeout,
+			self._timeout_callback, task_scheduler)
+		task_scheduler.addExitListener(self._exit_callback)
 
 		try:
 			task_scheduler.start()
-			while not self._timed_out and task_scheduler.poll() is None:
-				event_loop.iteration()
-			if self._timed_out:
-				task_scheduler.cancel()
+			event_loop.run_until_complete(self._run_done)
 			task_scheduler.wait()
 		finally:
 			event_loop.source_remove(timeout_id)
diff --git a/pym/portage/util/_eventloop/EventLoop.py b/pym/portage/util/_eventloop/EventLoop.py
index 8095400..8f13de3 100644
--- a/pym/portage/util/_eventloop/EventLoop.py
+++ b/pym/portage/util/_eventloop/EventLoop.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2014 Gentoo Foundation
+# Copyright 1999-2016 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import division
@@ -577,6 +577,21 @@ class EventLoop(object):
 		del self._poll_event_handlers[f]
 		return True
 
+	def run_until_complete(self, future):
+		"""
+		Run until the Future is done.
+
+		@type future: asyncio.Future
+		@param future: a Future to wait for
+		@rtype: object
+		@return: the Future's result
+		@raise: the Future's exception
+		"""
+		while not future.done():
+			self.iteration()
+
+		return future.result()
+
 _can_poll_device = None
 
 def can_poll_device():
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [gentoo-portage-dev] [PATCH] EventLoop: add run_until_complete method (bug 591760)
  2016-08-20 22:18 [gentoo-portage-dev] [PATCH] EventLoop: add run_until_complete method (bug 591760) Zac Medico
@ 2016-08-22 10:18 ` Alexander Berntsen
  2016-08-22 16:04   ` Zac Medico
  2016-08-22 16:34 ` [gentoo-portage-dev] [PATCH v2] " Zac Medico
  1 sibling, 1 reply; 6+ messages in thread
From: Alexander Berntsen @ 2016-08-22 10:18 UTC (permalink / raw
  To: gentoo-portage-dev


[-- Attachment #1.1: Type: text/plain, Size: 540 bytes --]

On 21/08/16 00:18, Zac Medico wrote:
> This emulates the
> asyncio.AbstractEventLoop.run_until_complete(future) interface
What's that, and why are we emulating it instead of just using it?

> which will make it possible to reduce latency in situations where it
> is desirable for a loop to exit at the earliest opportunity.
It makes it possible -- okay. But does it? I'm reluctant to change
things willy nilly, and add SLOCs, without a tangible benefit.
-- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [gentoo-portage-dev] [PATCH] EventLoop: add run_until_complete method (bug 591760)
  2016-08-22 10:18 ` Alexander Berntsen
@ 2016-08-22 16:04   ` Zac Medico
  0 siblings, 0 replies; 6+ messages in thread
From: Zac Medico @ 2016-08-22 16:04 UTC (permalink / raw
  To: gentoo-portage-dev

On 08/22/2016 03:18 AM, Alexander Berntsen wrote:
> On 21/08/16 00:18, Zac Medico wrote:
>> This emulates the
>> asyncio.AbstractEventLoop.run_until_complete(future) interface
> What's that, and why are we emulating it instead of just using it?

We will be able to use it with python 3.4 and later, but not with older
versions of python. Also, we have some more work to do first:

* Migrate all internal use of the EventLoop.iteration method to the new
run_until_complete(future) method, and remove the EventLoop.iteration
method (or make it private as long as it's needed to implement
run_until_complete for older python versions).

* Implement all EventLoop methods using asyncio.AbstractEventLoop
methods (but keep existing implementations for use with older python).

>> which will make it possible to reduce latency in situations where it
>> is desirable for a loop to exit at the earliest opportunity.
> It makes it possible -- okay. But does it? I'm reluctant to change
> things willy nilly, and add SLOCs, without a tangible benefit.

Having a migration path to asyncio is the most tangible benefit, since
it allows us to rely on a standard library instead of our own internal
event loop implementation. I mention latency since it's a clear
advantage of the run_until_complete API (at least in theory).
-- 
Thanks,
Zac


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [gentoo-portage-dev] [PATCH v2] EventLoop: add run_until_complete method (bug 591760)
  2016-08-20 22:18 [gentoo-portage-dev] [PATCH] EventLoop: add run_until_complete method (bug 591760) Zac Medico
  2016-08-22 10:18 ` Alexander Berntsen
@ 2016-08-22 16:34 ` Zac Medico
  2016-08-23 15:40   ` Alexander Berntsen
  1 sibling, 1 reply; 6+ messages in thread
From: Zac Medico @ 2016-08-22 16:34 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Zac Medico

This emulates the asyncio.AbstractEventLoop.run_until_complete(future)
interface, which will make it possible to reduce latency in situations
where it is desirable for a loop to exit at the earliest opportunity.

The most tangible benefit of this change is that it provides a
migration path to asyncio, which will allow us to rely on a standard
library instead of our own internal event loop implementation.

In order to migrate to asyncio, more work is planned:

* Migrate all internal use of the EventLoop.iteration method to the new
run_until_complete(future) method, and remove the EventLoop.iteration
method (or make it private as long as it's needed to implement
run_until_complete for older python versions).

* Implement all EventLoop methods using asyncio.AbstractEventLoop
methods (but keep existing implementations for use with older python).

X-Gentoo-bug: 591760
X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760
---
[PATCH v2] only updates the commit message to provide more context

 pym/portage/tests/ebuild/test_ipc_daemon.py | 23 ++++++++++++++---------
 pym/portage/util/_eventloop/EventLoop.py    | 17 ++++++++++++++++-
 2 files changed, 30 insertions(+), 10 deletions(-)

diff --git a/pym/portage/tests/ebuild/test_ipc_daemon.py b/pym/portage/tests/ebuild/test_ipc_daemon.py
index 835f51f..68f139a 100644
--- a/pym/portage/tests/ebuild/test_ipc_daemon.py
+++ b/pym/portage/tests/ebuild/test_ipc_daemon.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2015 Gentoo Foundation
+# Copyright 2010-2016 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 import tempfile
@@ -16,6 +16,7 @@ from portage.util import ensure_dirs
 from portage.util._async.ForkProcess import ForkProcess
 from portage.util._async.TaskScheduler import TaskScheduler
 from portage.util._eventloop.global_event_loop import global_event_loop
+from portage.util.futures.futures import Future
 from _emerge.SpawnProcess import SpawnProcess
 from _emerge.EbuildBuildDir import EbuildBuildDir
 from _emerge.EbuildIpcDaemon import EbuildIpcDaemon
@@ -140,19 +141,23 @@ class IpcDaemonTestCase(TestCase):
 				build_dir.unlock()
 			shutil.rmtree(tmpdir)
 
-	def _timeout_callback(self):
-		self._timed_out = True
+	def _timeout_callback(self, task_scheduler):
+		task_scheduler.cancel()
+		self._exit_callback(task_scheduler)
+
+	def _exit_callback(self, task_scheduler):
+		if not self._run_done.done():
+			self._run_done.set_result(True)
 
 	def _run(self, event_loop, task_scheduler, timeout):
-		self._timed_out = False
-		timeout_id = event_loop.timeout_add(timeout, self._timeout_callback)
+		self._run_done = Future()
+		timeout_id = event_loop.timeout_add(timeout,
+			self._timeout_callback, task_scheduler)
+		task_scheduler.addExitListener(self._exit_callback)
 
 		try:
 			task_scheduler.start()
-			while not self._timed_out and task_scheduler.poll() is None:
-				event_loop.iteration()
-			if self._timed_out:
-				task_scheduler.cancel()
+			event_loop.run_until_complete(self._run_done)
 			task_scheduler.wait()
 		finally:
 			event_loop.source_remove(timeout_id)
diff --git a/pym/portage/util/_eventloop/EventLoop.py b/pym/portage/util/_eventloop/EventLoop.py
index 8095400..8f13de3 100644
--- a/pym/portage/util/_eventloop/EventLoop.py
+++ b/pym/portage/util/_eventloop/EventLoop.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2014 Gentoo Foundation
+# Copyright 1999-2016 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import division
@@ -577,6 +577,21 @@ class EventLoop(object):
 		del self._poll_event_handlers[f]
 		return True
 
+	def run_until_complete(self, future):
+		"""
+		Run until the Future is done.
+
+		@type future: asyncio.Future
+		@param future: a Future to wait for
+		@rtype: object
+		@return: the Future's result
+		@raise: the Future's exception
+		"""
+		while not future.done():
+			self.iteration()
+
+		return future.result()
+
 _can_poll_device = None
 
 def can_poll_device():
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [gentoo-portage-dev] [PATCH v2] EventLoop: add run_until_complete method (bug 591760)
  2016-08-22 16:34 ` [gentoo-portage-dev] [PATCH v2] " Zac Medico
@ 2016-08-23 15:40   ` Alexander Berntsen
  2016-08-23 16:51     ` Zac Medico
  0 siblings, 1 reply; 6+ messages in thread
From: Alexander Berntsen @ 2016-08-23 15:40 UTC (permalink / raw
  To: gentoo-portage-dev


[-- Attachment #1.1: Type: text/plain, Size: 90 bytes --]

OK, LGTM.

-- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [gentoo-portage-dev] [PATCH v2] EventLoop: add run_until_complete method (bug 591760)
  2016-08-23 15:40   ` Alexander Berntsen
@ 2016-08-23 16:51     ` Zac Medico
  0 siblings, 0 replies; 6+ messages in thread
From: Zac Medico @ 2016-08-23 16:51 UTC (permalink / raw
  To: gentoo-portage-dev

On 08/23/2016 08:40 AM, Alexander Berntsen wrote:
> OK, LGTM.
> 

Pushed:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=995f0f983386e2a82dbce65d4366ee7f58f59138
-- 
Thanks,
Zac


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2016-08-23 16:51 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-08-20 22:18 [gentoo-portage-dev] [PATCH] EventLoop: add run_until_complete method (bug 591760) Zac Medico
2016-08-22 10:18 ` Alexander Berntsen
2016-08-22 16:04   ` Zac Medico
2016-08-22 16:34 ` [gentoo-portage-dev] [PATCH v2] " Zac Medico
2016-08-23 15:40   ` Alexander Berntsen
2016-08-23 16:51     ` Zac Medico

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox