From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id BAF39138A87 for ; Mon, 23 Feb 2015 20:59:23 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 6F03AE08BA; Mon, 23 Feb 2015 20:59:22 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 9F4C8E08A4 for ; Mon, 23 Feb 2015 20:59:21 +0000 (UTC) Received: from oystercatcher.gentoo.org (oystercatcher.gentoo.org [148.251.78.52]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id BB30734093B for ; Mon, 23 Feb 2015 20:59:20 +0000 (UTC) Received: from localhost.localdomain (localhost [127.0.0.1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id 5B92F12678 for ; Mon, 23 Feb 2015 20:59:19 +0000 (UTC) From: "Alex Legler" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Alex Legler" Message-ID: <1424725085.88d9781c469e27ca77901a05326f598821715b33.a3li@gentoo> Subject: [gentoo-commits] proj/ag:master commit in: /, lib/ X-VCS-Repository: proj/ag X-VCS-Files: ag lib/storage.rb lib/utils.rb X-VCS-Directories: / lib/ X-VCS-Committer: a3li X-VCS-Committer-Name: Alex Legler X-VCS-Revision: 88d9781c469e27ca77901a05326f598821715b33 X-VCS-Branch: master Date: Mon, 23 Feb 2015 20:59:19 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Archives-Salt: fc5a4744-1486-4095-b456-e23e0110d133 X-Archives-Hash: f55a0d18c8dac973ba3e6163fbfc7293 commit: 88d9781c469e27ca77901a05326f598821715b33 Author: Alex Legler a3li li> AuthorDate: Mon Feb 23 20:58:05 2015 +0000 Commit: Alex Legler gentoo org> CommitDate: Mon Feb 23 20:58:05 2015 +0000 URL: http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=88d9781c Only occupy 3/4 of the processors --- ag | 6 +++--- lib/storage.rb | 6 +++--- lib/utils.rb | 8 ++++++++ 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/ag b/ag index 73c189b..bcb1ac6 100755 --- a/ag +++ b/ag @@ -115,13 +115,13 @@ def do_full messages = $maildir.list(:cur) - Parallel.each(messages, progress: "Importing #{$options.name}") do |maildir_message| + Parallel.each(messages, in_processes: Ag::Utils.proc_count, progress: "Importing #{$options.name}") do |maildir_message| mail = maildir_message.data begin Ag::Storage.store($options.name, mail, maildir_message.unique_name) rescue => e - $stderr.puts "Cannot save message #{mail.message_id}: #{e.message}" + $stderr.puts "Cannot save message #{mail.message_id}: (#{e.class}) #{e.message}" if $options.debug next end end @@ -132,7 +132,7 @@ end def do_incremental messages = $maildir.list(:cur) - Parallel.each(messages, progress: "Importing #{$options.name}") do |maildir_message| + Parallel.each(messages, in_processes: Ag::Utils.proc_count, progress: "Importing #{$options.name}") do |maildir_message| mail = maildir_message.data begin diff --git a/lib/storage.rb b/lib/storage.rb index 6409df2..5b360f5 100644 --- a/lib/storage.rb +++ b/lib/storage.rb @@ -201,9 +201,9 @@ module Ag::Storage def fix_threading(list, pass) result = $es.search( index: 'ml-' + list, - size: 100000, + size: 5000, body: { - size: 100000, + size: 5000, query: { filtered: { filter: { @@ -225,7 +225,7 @@ module Ag::Storage } ) - Parallel.each(result['hits']['hits'], progress: "Calculating Threading (Pass #{pass})") do |hit| + Parallel.each(result['hits']['hits'], in_processes: Ag::Utils.proc_count, progress: "Calculating Threading (Pass #{pass})") do |hit| msg = resolve_message_id(list, hit['_source']['raw_parent']) unless msg == nil diff --git a/lib/utils.rb b/lib/utils.rb index d621a2e..0213c6d 100644 --- a/lib/utils.rb +++ b/lib/utils.rb @@ -1,5 +1,9 @@ require 'charlock_holmes' +module Parallel::ProcessorCount + module_function :processor_count +end + module Ag module Utils @@ -62,5 +66,9 @@ module Ag rescue ArgumentError '' end + + def proc_count + (Parallel::ProcessorCount.processor_count.to_f * 0.75).floor + end end end \ No newline at end of file