[gentoo-commits] proj/ag:master commit in: lib/

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     d9e1049bd00c469805779257343b049ff45ff0fe
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Sat Feb 21 21:35:37 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Sat Feb 21 21:35:37 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=d9e1049b

Sleep for a second after creating an index to hopefully fix timing issues

---
 lib/storage.rb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/storage.rb b/lib/storage.rb
index 9045a0b..da9e8ad 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -68,6 +68,9 @@ module Ag::Storage
           }
         }
       })
+
+    # Give elasticsearch some time to process the new index
+    sleep 1
   end
 
   def get_content(message)

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     5b1feb7c720a04b5409a2da76fb6b95b6ee83955
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Sun Feb 22 23:01:10 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Sun Feb 22 23:01:10 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=5b1feb7c

Save message filename in the index

---
 lib/storage.rb | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index fdf70b6..f4e55b6 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -55,6 +55,10 @@ module Ag::Storage
                 type: 'string',
                 index: 'not_analyzed'
               },
+              raw_filename: {
+                type: 'string',
+                index: 'not_analyzed'
+              },
               raw_parent: {
                 type: 'string'
               },
@@ -157,7 +161,8 @@ module Ag::Storage
         month: ("%i%02i" % [message.date.year, message.date.month]).to_i, # this is a sortable number!
         content: content,
         attachments: attachments,
-        raw_parent: raw_parent
+        raw_parent: raw_parent,
+        raw_filename: filename
       }
     )
   end

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     215091d5d1abeb9c38540eb0bb569851a3de1ba6
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Sun Feb 22 23:50:38 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Sun Feb 22 23:50:38 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=215091d5

Refactor id resolving method to a more generic function

---
 lib/storage.rb | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 660b300..37083fa 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -93,8 +93,8 @@ module Ag::Storage
     content
   end
 
-  def resolve_message_id(list, message_id = nil)
-    return nil if message_id == nil
+  def resolve_by_field(list, field, value)
+    return nil if value == nil
 
     result = $es.search(
       index: 'ml-' + list,
@@ -102,7 +102,7 @@ module Ag::Storage
         query: {
           filtered: {
             filter: {
-              term: { raw_message_id: message_id }
+              term: { field => value }
             }
           }
         },
@@ -111,10 +111,17 @@ module Ag::Storage
     )
 
     return nil if result['hits']['total'] == 0
-
     result['hits']['hits'].first['_id']
   end
 
+  def resolve_message_id(list, message_id = nil)
+    resolve_by_field(list, :raw_message_id, message_id)
+  end
+
+  def resolve_filename(list, filename)
+    resolve_by_field(list, :raw_filename, filename)
+  end
+
   def store(list, message, filename)
     content = get_content(message, filename)
 

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     38b11468dafd19db7793525152710ce824f24ccc
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Sun Feb 22 23:05:43 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Sun Feb 22 23:05:43 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=38b11468

Give message-id to hash function an appropriate name

---
 lib/storage.rb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index f4e55b6..b4873d8 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -93,7 +93,7 @@ module Ag::Storage
     content
   end
 
-  def get_parent_message(list, parent_message_id = nil)
+  def resolve_message_id(list, parent_message_id = nil)
     return nil if parent_message_id == nil
 
     result = $es.search(
@@ -195,7 +195,7 @@ module Ag::Storage
     )
 
     result['hits']['hits'].each do |hit|
-      msg = get_parent_message(list, hit['_source']['raw_parent'])
+      msg = resolve_message_id(list, hit['_source']['raw_parent'])
 
       unless msg == nil
         $es.update(

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     ccfb2c3d3f4f78a21dd5f15a36450b89da8b8329
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Mon Feb 23 00:04:43 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Mon Feb 23 00:04:43 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=ccfb2c3d

Add id resolving helper

---
 lib/utils.rb | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/lib/utils.rb b/lib/utils.rb
index 3714614..aaa04c7 100644
--- a/lib/utils.rb
+++ b/lib/utils.rb
@@ -14,5 +14,17 @@ module Ag
         'Encoding could not be reliably detected. Contents not available.'
       end
     end
+
+    def resolve_id
+      id = $options.dir
+      case $options.argmode
+      when :msgid
+        id = Ag::Storage.resolve_message_id($options.name, $options.dir)
+      when :file
+        id = Ag::Storage.resolve_filename($options.name, $options.dir)
+      end
+
+      id
+    end
   end
 end
\ No newline at end of file

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     2ebbd8d3908a6095aa4a1049b8c3e02b4ca979a6
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Sun Feb 22 23:11:34 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Sun Feb 22 23:11:34 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=2ebbd8d3

also rename parameter name

---
 lib/storage.rb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index b4873d8..6bbb527 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -93,8 +93,8 @@ module Ag::Storage
     content
   end
 
-  def resolve_message_id(list, parent_message_id = nil)
-    return nil if parent_message_id == nil
+  def resolve_message_id(list, message_id = nil)
+    return nil if message_id == nil
 
     result = $es.search(
       index: 'ml-' + list,
@@ -102,7 +102,7 @@ module Ag::Storage
         query: {
           filtered: {
             filter: {
-              term: { raw_message_id: parent_message_id }
+              term: { raw_message_id: message_id }
             }
           }
         },

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     95040b1680f96e5391902de301cf07f3c2c6d571
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Sun Feb 22 23:50:12 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Sun Feb 22 23:50:12 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=95040b16

Fix get_content invocation for new error messages

---
 lib/storage.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 6bbb527..660b300 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -116,7 +116,7 @@ module Ag::Storage
   end
 
   def store(list, message, filename)
-    content = get_content(message)
+    content = get_content(message, filename)
 
     identifier = message['X-Archives-Hash'].value
     raw_parent = Ag::Threading.get_parent_message_id(message)

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     bbc8b2a61544af2859276cf8badb53ca3b1244c5
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Mon Feb 23 20:56:21 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Mon Feb 23 20:56:21 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=bbc8b2a6

Test (hopefully) faster encoding fix method

---
 lib/utils.rb | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/utils.rb b/lib/utils.rb
index aaa04c7..fc4427e 100644
--- a/lib/utils.rb
+++ b/lib/utils.rb
@@ -2,8 +2,15 @@ require 'charlock_holmes'
 
 module Ag
   module Utils
+
     module_function
-    def fix_encoding(str, fail_hard = false)
+    def fix_encoding(str)
+      s = str.encode('UTF-8', 'UTF-8', invalid: :replace, replace: '')
+      s = s.unpack('C*').pack('U*') unless s.valid_encoding?
+      s
+    end
+
+    def fix_encoding_old(str, fail_hard = false)
       detection = CharlockHolmes::EncodingDetector.detect(str)
       CharlockHolmes::Converter.convert(str, detection[:encoding], 'UTF-8')
     rescue => e

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     d38a68108d6224c7d5e8fd97fcdfe26ef0e0cf8c
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Mon Feb 23 20:57:35 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Mon Feb 23 20:57:35 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=d38a6810

More robust header parsing (again hopefully)

---
 lib/storage.rb | 54 +++++++++++++++++++++++++++++++++++++++---------------
 lib/utils.rb   | 29 +++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+), 15 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index d32ba2b..6409df2 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -28,6 +28,18 @@ module Ag::Storage
                   }
                 }
               },
+              received: {
+                properties: {
+                  hop: {
+                    type: 'string',
+                    index: 'not_analyzed'
+                  },
+                  date: {
+                    type: 'date',
+                    format: 'dateOptionalTime'
+                  }
+                }
+              },
               cc: {
                 type: 'string'
               },
@@ -81,7 +93,7 @@ module Ag::Storage
     content = 'Cannot parse MIME/contents.'
     begin
       raw_content = Ag::Rendering::HTMLizer.HTMLize(message)
-      content = Ag::Utils.fix_encoding(raw_content || '', true).strip
+      content = Ag::Utils.fix_encoding(raw_content || '').strip
 
       if content == ''
         $stderr.puts "#{message.message_id}: Content empty?" if $options.debug
@@ -125,24 +137,35 @@ module Ag::Storage
   def store(list, message, filename)
     content = get_content(message, filename)
 
-    identifier = message['X-Archives-Hash'].value
+    identifier = nil
+    begin
+      identifier = message['X-Archives-Hash'].value
+    rescue NoMethodError
+      raise 'No archives hash'
+    end
+
     raw_parent = Ag::Threading.get_parent_message_id(message)
 
-    from = Ag::Utils.fix_encoding(message[:from].formatted.first)
-    from_realname = from.gsub(/<(.*)>/, '').strip
+    from = Ag::Utils.resolve_address_header(message, :from).first
+    from_realname = Ag::Utils.get_sender_displayname(message)
+    to = Ag::Utils.resolve_address_header(message, :to)
+    cc = Ag::Utils.resolve_address_header(message, :cc)
+    subject = Ag::Utils.fix_encoding(message.subject)
 
-    to = ''
-    if message[:to]
-      to = Ag::Utils.fix_encoding(message[:to].formatted.join(','))
-    end
+    date = [message.received].flatten.first.field.date_time
 
-    cc = ''
-    if message[:cc]
-      cc = Ag::Utils.fix_encoding(message[:cc].formatted.join(','))
+    received = []
+    [message.received].flatten.each do |hop|
+      begin
+        received << {
+          hop: hop.field.info,
+          date: hop.field.date_time
+        }
+      rescue => e
+        next
+      end
     end
 
-    subject = Ag::Utils.fix_encoding(message.subject)
-
     attachments = []
     if message.has_attachments?
       message.attachments.each do |attachment|
@@ -164,10 +187,11 @@ module Ag::Storage
         cc: cc,
         from: from,
         from_realname: from_realname,
-        date: message.date,
-        month: ("%i%02i" % [message.date.year, message.date.month]).to_i, # this is a sortable number!
+        date: date,
+        month: ("%i%02i" % [date.year, date.month]).to_i, # this is a sortable number!
         content: content,
         attachments: attachments,
+        received: received,
         raw_parent: raw_parent,
         raw_filename: filename
       }

diff --git a/lib/utils.rb b/lib/utils.rb
index fc4427e..d621a2e 100644
--- a/lib/utils.rb
+++ b/lib/utils.rb
@@ -33,5 +33,34 @@ module Ag
 
       id
     end
+
+    def resolve_address_header(message, header)
+      if message[header].is_a? Mail::StructuredField
+        # Good header, properly parsed
+        message[header].addrs.map {|s| fix_encoding(s.to_s)}
+      elsif nil == message[header]
+        # Header not set, return empty
+        []
+      else
+        # Parsing failed, do best-effort parsing
+        [message[header].to_s.split(/,\s*/)].flatten.map {|s| fix_encoding(s)}
+      end
+    rescue ArgumentError
+      []
+    end
+
+    def get_sender_displayname(message)
+      if message[:from].is_a? Mail::StructuredField
+        fix_encoding(message[:from].addrs.first.to_s).strip
+      else
+        if message[:from].respond_to? :addrs and display_name = message[:from].addrs.first.display_name
+          fix_encoding(display_name).strip
+        else
+          fix_encoding(message[:from].to_s).strip
+        end
+      end
+    rescue ArgumentError
+      ''
+    end
   end
 end
\ No newline at end of file

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     c279eb2a9cf679731d643904c07d35bbc41d2dc7
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Mon Feb 23 23:54:34 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Mon Feb 23 23:55:20 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=c279eb2a

Fix displayname extraction

---
 lib/utils.rb | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/lib/utils.rb b/lib/utils.rb
index b37437b..f8d546e 100644
--- a/lib/utils.rb
+++ b/lib/utils.rb
@@ -54,14 +54,10 @@ module Ag
     end
 
     def get_sender_displayname(message)
-      if message[:from].is_a? Mail::StructuredField
-        fix_encoding(message[:from].addrs.first.to_s).strip
-      else
-        if message[:from].respond_to? :addrs and display_name = message[:from].addrs.first.display_name
-          fix_encoding(display_name).strip
-        else
-          fix_encoding(message[:from].to_s).strip
-        end
+      begin
+        fix_encoding(message[:from].addrs.first.display_name).strip
+      rescue NoMethodError
+        fix_encoding(message[:from].to_s).strip
       end
     rescue ArgumentError
       ''

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     d42179d8d444cc35698e06d2d7520e6f7b383f17
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Mon Feb 23 23:54:59 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Mon Feb 23 23:55:20 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=d42179d8

Fall back to charlock_holmes; also add stub encoding extraction from the message for later on

---
 lib/rendering.rb | 22 ++++++++++++++++------
 lib/utils.rb     |  4 ++--
 2 files changed, 18 insertions(+), 8 deletions(-)

diff --git a/lib/rendering.rb b/lib/rendering.rb
index 3e77414..7649fcf 100644
--- a/lib/rendering.rb
+++ b/lib/rendering.rb
@@ -8,33 +8,43 @@ module Ag::Rendering
         content_type = mime_split(mail.parts.first.content_type)
 
         if content_type == 'text/plain' or content_type == 'text/html'
-          to_content(content_type, mail.parts.first.decoded)
+          to_content(content_type, mail.parts.first.decoded, get_encoding(mail.parts.first))
         else
           # Nested multipart?
           if mail.parts.first.multipart?
             content_type = mime_split(mail.parts.first.parts.first.content_type)
 
             if content_type == 'text/plain' or content_type == 'text/html'
-              to_content(content_type, mail.parts.first.parts.first.decoded)
+              to_content(content_type, mail.parts.first.parts.first.decoded, get_encoding(mail.parts.first.parts.first))
             else
               raise "Cannot find body: #{mail.message_id}"
             end
           # Specialty: Gnus/Emacs signed emails with no explicit multipart type
           elsif mime_split(mail.content_type) == 'multipart/signed'
-            to_content('text/plain', mail.parts.first.decoded)
+            to_content('text/plain', mail.parts.first.decoded, get_encoding(mail.parts.first))
           end
         end
       else
         # No Content-Type, assume plain text (git-send-email)
         if mail.content_type == nil
-          to_content('text/plain', mail.body.decoded)
+          to_content('text/plain', mail.body.decoded, get_encoding(mail))
         else
-          to_content(mime_split(mail.content_type), mail.body.decoded)
+          to_content(mime_split(mail.content_type), mail.body.decoded, get_encoding(mail))
         end
       end
     end
 
-    def self.to_content(content_type, content)
+    def self.get_encoding(part)
+      if part.content_type_parameters
+        part.content_type_parameters['charset']
+      else
+        nil
+      end
+    end
+
+    def self.to_content(content_type, content, charset = nil)
+      #content = content.force_encoding(charset) if charset
+
       if content_type == 'text/plain'
         escaped_content = CGI::escapeHTML(content)
         escaped_content.lines.map do |line|

diff --git a/lib/utils.rb b/lib/utils.rb
index f8d546e..38349e0 100644
--- a/lib/utils.rb
+++ b/lib/utils.rb
@@ -8,13 +8,13 @@ module Ag
   module Utils
 
     module_function
-    def fix_encoding(str)
+    def fix_encoding2(str)
       s = str.encode('UTF-8', 'UTF-8', invalid: :replace, replace: '')
       s = s.unpack('C*').pack('U*') unless s.valid_encoding?
       s
     end
 
-    def fix_encoding_old(str, fail_hard = false)
+    def fix_encoding(str, fail_hard = false)
       detection = CharlockHolmes::EncodingDetector.detect(str)
       CharlockHolmes::Converter.convert(str, detection[:encoding], 'UTF-8')
     rescue => e

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     92a7b7822e1f65a53b6d1ed66f3fc0b73c639f0d
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 02:49:50 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 02:49:50 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=92a7b782

Improve sleep code.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index c054f9e..bf90202 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -87,8 +87,10 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
-	puts $es.cluster.health
-    sleep 1
+    while $es.cluster.health['status'] != 'green' do
+        sleep 0.05
+        puts $es.cluster.health
+    end
   end
 
   def get_content(message, filename)

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     d83a8a9a7ab8e204ef5ba0161a6424e854a28a93
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 02:52:49 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 02:52:49 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=d83a8a9a

Improve poll code.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index bf90202..fa1fa51 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -11,8 +11,9 @@ module Ag::Storage
   end
 
   def create_index(list)
+    indexname = 'ml-' + list
     $es.indices.create(
-      index: 'ml-' + list,
+      index: indexname,
       body: {
         mappings: {
           message: {
@@ -87,9 +88,12 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
+    puts $es.cluster.health
+    puts $es.indices.status(index: indexname)
     while $es.cluster.health['status'] != 'green' do
-        sleep 0.05
         puts $es.cluster.health
+		puts $es.indices.status(index: indexname)
+        sleep 0.05
     end
   end
 

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     1b436d3f55805bd0e7ae7f19eb5e87c08bd6d36c
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 02:53:49 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 02:53:49 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=1b436d3f

This is hard to make it trip up sometimes.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 2 --
 1 file changed, 2 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index fa1fa51..84e1f8f 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -88,10 +88,8 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
-    puts $es.cluster.health
     puts $es.indices.status(index: indexname)
     while $es.cluster.health['status'] != 'green' do
-        puts $es.cluster.health
 		puts $es.indices.status(index: indexname)
         sleep 0.05
     end

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     10afa5d46e888e67cf41bb87c6cf82fa2df30d69
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 02:55:15 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 02:55:15 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=10afa5d4

Hard to read.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 84e1f8f..78c83a2 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -88,10 +88,10 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
-    puts $es.indices.status(index: indexname)
+    pp $es.indices.status(index: indexname)
     while $es.cluster.health['status'] != 'green' do
-		puts $es.indices.status(index: indexname)
-        sleep 0.05
+		pp $es.indices.status(index: indexname)
+        sleep 0.01
     end
   end
 

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     141b4a89ef5409f443c77d736ad7c2d67b410296
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 02:57:50 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 02:57:50 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=141b4a89

missing dep.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/storage.rb b/lib/storage.rb
index 78c83a2..66e3ba4 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -88,6 +88,7 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
+    require 'pp'
     pp $es.indices.status(index: indexname)
     while $es.cluster.health['status'] != 'green' do
 		pp $es.indices.status(index: indexname)

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     85ae5841baf289f178cfb573a6920a080b6cdb55
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 02:59:14 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 02:59:14 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=85ae5841

Fun to trigger race bugs.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 66e3ba4..f47c6bc 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -1,5 +1,6 @@
 require 'elasticsearch'
 require 'date'
+require 'pp'
 
 module Ag::Storage
   module_function
@@ -88,7 +89,6 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
-    require 'pp'
     pp $es.indices.status(index: indexname)
     while $es.cluster.health['status'] != 'green' do
 		pp $es.indices.status(index: indexname)

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     bd0a53c8bc9a1bb86908e15c1c0310593a09c14a
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 03:02:16 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 03:02:16 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=bd0a53c8

Work on getting state.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index f47c6bc..081e206 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -89,7 +89,10 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
-    pp $es.indices.status(index: indexname)
+	status = $es.indices.status(index: indexname)
+    pp status['indices'][indexname]['shares'].map do |k,v|
+		v['routing']['state']
+	end
     while $es.cluster.health['status'] != 'green' do
 		pp $es.indices.status(index: indexname)
         sleep 0.01

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     c4c504d20e2998932514deffa2e573310eb2fdf6
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 03:06:39 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 03:06:39 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=c4c504d2

Work on better selection of status.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index e9aea92..7cba45c 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -89,10 +89,12 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
-	status = $es.indices.status(index: indexname)
-    pp status['indices'][indexname]['shares'].map do |k,v|
-		v['routing']['state']
-	end
+    status = $es.indices.status(index: indexname)
+    pp status
+    status = status['indices'][indexname]['shards'].map do |k,v|
+        v['routing']['state']
+    end
+    pp status
     while $es.cluster.health['status'] != 'green' do
 		pp $es.indices.status(index: indexname)
         sleep 0.01

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     12fc88be4083d49752af89733a4d5ae89c715527
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 03:08:23 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 03:08:23 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=12fc88be

Key hell.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 7cba45c..29311ad 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -91,8 +91,9 @@ module Ag::Storage
     # Give elasticsearch some time to process the new index
     status = $es.indices.status(index: indexname)
     pp status
+    pp status['indices'][indexname]['shards']
     status = status['indices'][indexname]['shards'].map do |k,v|
-        v['routing']['state']
+        v[0]['routing']['state']
     end
     pp status
     while $es.cluster.health['status'] != 'green' do

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     e4394a608a896073bf0bf13483436e3cc04be5ee
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 03:16:51 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 03:16:51 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=e4394a60

Work on the index.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 29311ad..6e41ed7 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -89,17 +89,17 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
-    status = $es.indices.status(index: indexname)
-    pp status
-    pp status['indices'][indexname]['shards']
-    status = status['indices'][indexname]['shards'].map do |k,v|
-        v[0]['routing']['state']
-    end
-    pp status
     while $es.cluster.health['status'] != 'green' do
-		pp $es.indices.status(index: indexname)
-        sleep 0.01
+      status = $es.indices.status(index: indexname)
+      pp status
+      pp status['indices'][indexname]['shards']
+      status = status['indices'][indexname]['shards'].map do |k,v|
+        v[0]['routing']['state']
+      end
+      pp status
+      sleep 0.01
     end
+    pp $es.indices.status(index: indexname)
   end
 
   def get_content(message, filename)

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     e55ce719ed5c077913d1e9e39b4982e6ed34b472
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 03:19:07 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 03:19:07 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=e55ce719

more print.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 6e41ed7..8828bf6 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -89,8 +89,8 @@ module Ag::Storage
       })
 
     # Give elasticsearch some time to process the new index
+    status = $es.indices.status(index: indexname)
     while $es.cluster.health['status'] != 'green' do
-      status = $es.indices.status(index: indexname)
       pp status
       pp status['indices'][indexname]['shards']
       status = status['indices'][indexname]['shards'].map do |k,v|
@@ -98,6 +98,7 @@ module Ag::Storage
       end
       pp status
       sleep 0.01
+      status = $es.indices.status(index: indexname)
     end
     pp $es.indices.status(index: indexname)
   end

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     f18e7249b6615030030260b68f52cdf31f45e915
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 03:32:51 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 03:32:51 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=f18e7249

Almost ready for real state.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/lib/storage.rb b/lib/storage.rb
index 8828bf6..7ec0861 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -5,6 +5,21 @@ require 'pp'
 module Ag::Storage
   module_function
 
+  def index_states(indexname)
+    status = $es.indices.status(index: indexname)
+    states = status['indices'][indexname]['shards'].map do |key,array|
+      array.map do |v|
+          v['routing']['state']
+      end.flatten
+    end.flatten
+  end
+  def index_ready?(indexname)
+    states = index_states(indexname)
+    not_ready = states.include? 'INITIALIZING' or states.include? 'RELOCATING'
+    return !not_ready
+    pp $es.cluster.health(level: indices)
+  end
+
   # Throws Elasticsearch::Transport::Transport::Errors::NotFound
   # if the list does not exist
   def delete_index(list)
@@ -88,6 +103,7 @@ module Ag::Storage
         }
       })
 
+    pp $es.cluster.health(level: indices)
     # Give elasticsearch some time to process the new index
     status = $es.indices.status(index: indexname)
     while $es.cluster.health['status'] != 'green' do

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     51aea5a8f68dc5d4a49e6795ff83d439c7f03942
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 03:39:44 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 03:39:44 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=51aea5a8

Working index status check!

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 37 +++++++++++--------------------------
 1 file changed, 11 insertions(+), 26 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 7ec0861..4135971 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -5,28 +5,25 @@ require 'pp'
 module Ag::Storage
   module_function
 
-  def index_states(indexname)
-    status = $es.indices.status(index: indexname)
-    states = status['indices'][indexname]['shards'].map do |key,array|
-      array.map do |v|
-          v['routing']['state']
-      end.flatten
-    end.flatten
+  def index_status(indexname)
+    $es.cluster.health(level: :indices)['indices'][indexname]['status']
   end
+
   def index_ready?(indexname)
-    states = index_states(indexname)
-    not_ready = states.include? 'INITIALIZING' or states.include? 'RELOCATING'
-    return !not_ready
-    pp $es.cluster.health(level: indices)
+    status = index_status(indexname) 
+    # if you call index_status twice, you might get different results!
+    return status == 'green' or status == 'yellow'
   end
 
   # Throws Elasticsearch::Transport::Transport::Errors::NotFound
   # if the list does not exist
   def delete_index(list)
-      $es.indices.delete index: 'ml-' + list
+      $es.indices.delete(index: 'ml-' + list)
   end
 
-  def create_index(list)
+  # Create an index for list
+  # sleep in 5ms intervals until it is ready
+  def create_index(list, sleeptime: 0.005)
     indexname = 'ml-' + list
     $es.indices.create(
       index: indexname,
@@ -103,20 +100,8 @@ module Ag::Storage
         }
       })
 
-    pp $es.cluster.health(level: indices)
     # Give elasticsearch some time to process the new index
-    status = $es.indices.status(index: indexname)
-    while $es.cluster.health['status'] != 'green' do
-      pp status
-      pp status['indices'][indexname]['shards']
-      status = status['indices'][indexname]['shards'].map do |k,v|
-        v[0]['routing']['state']
-      end
-      pp status
-      sleep 0.01
-      status = $es.indices.status(index: indexname)
-    end
-    pp $es.indices.status(index: indexname)
+    sleep sleeptime while not index_ready?(indexname)
   end
 
   def get_content(message, filename)

[gentoo-commits] proj/ag:master commit in: lib/

[Robin H. Johnson]

commit:     1efec8d12b9cab3b706741dd2cd233d702ca4476
Author:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 24 03:42:21 2015 +0000
Commit:     Robin H. Johnson <robbat2 <AT> gentoo <DOT> org>
CommitDate: Tue Feb 24 03:42:21 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=1efec8d1

Cleanup check.

Signed-off-by: Robin H. Johnson <robbat2 <AT> gentoo.org>

---
 lib/storage.rb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 4135971..fe50ea8 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -6,13 +6,13 @@ module Ag::Storage
   module_function
 
   def index_status(indexname)
-    $es.cluster.health(level: :indices)['indices'][indexname]['status']
+    $es.cluster.health(level: :indices)['indices'][indexname]['status'] || 'FAIL'
   end
 
   def index_ready?(indexname)
-    status = index_status(indexname) 
+    # watch out if you rewrite this
     # if you call index_status twice, you might get different results!
-    return status == 'green' or status == 'yellow'
+    [:green, :yellow].include? index_status(indexname).downcase.to_sym
   end
 
   # Throws Elasticsearch::Transport::Transport::Errors::NotFound

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     ef30ac17210f8c5aeb9041faf1a4c34eaca5149e
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Wed Feb 25 10:31:03 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Wed Feb 25 10:31:03 2015 +0000
URL:        http://sources.gentoo.org/gitweb/?p=proj/ag.git;a=commit;h=ef30ac17

tab fix

---
 lib/storage.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index fe50ea8..e8de874 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -247,7 +247,7 @@ module Ag::Storage
     opts = {
         :in_processes => Ag::Utils.proc_count,
     }
-	opts[:progress] = "Calculating Threading (Pass #{pass})" if $options.progress
+    opts[:progress] = "Calculating Threading (Pass #{pass})" if $options.progress
     Parallel.each(result['hits']['hits'], opts) do |hit|
       msg = resolve_message_id(list, hit['_source']['raw_parent'])
 

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     b67e866d53ee3fd665d1d23ab69d609035cedca8
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Sat Mar  7 21:11:01 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Sat Mar  7 21:11:01 2015 +0000
URL:        https://gitweb.gentoo.org/proj/ag.git/commit/?id=b67e866d

one more minor fix

 lib/storage.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index e8de874..56f3ef0 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -15,7 +15,7 @@ module Ag::Storage
     [:green, :yellow].include? index_status(indexname).downcase.to_sym
   end
 
-  # Throws Elasticsearch::Transport::Transport::Errors::NotFound
+  # throws Elasticsearch::Transport::Transport::Errors::NotFound
   # if the list does not exist
   def delete_index(list)
       $es.indices.delete(index: 'ml-' + list)

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     89afb65f423c9eb3e34c09596603e60fb8a96729
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Sat Mar  7 21:13:50 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Sat Mar  7 21:13:50 2015 +0000
URL:        https://gitweb.gentoo.org/proj/ag.git/commit/?id=89afb65f

Clarify error message

 lib/storage.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/storage.rb b/lib/storage.rb
index 56f3ef0..6ba4026 100644
--- a/lib/storage.rb
+++ b/lib/storage.rb
@@ -160,7 +160,7 @@ module Ag::Storage
     begin
       identifier = message['X-Archives-Hash'].value
     rescue NoMethodError
-      raise 'No archives hash'
+      raise 'No archives hash found in the message headers'
     end
 
     raw_parent = Ag::Threading.get_parent_message_id(message)

[gentoo-commits] proj/ag:master commit in: lib/

[Alex Legler]

commit:     71ddec6c31a89d7b672f1827a2d87aea10c019f5
Author:     Alex Legler <alex <AT> a3li <DOT> li>
AuthorDate: Mon Jun 29 19:44:54 2015 +0000
Commit:     Alex Legler <a3li <AT> gentoo <DOT> org>
CommitDate: Mon Jun 29 19:44:54 2015 +0000
URL:        https://gitweb.gentoo.org/proj/ag.git/commit/?id=71ddec6c

Fix broken names when no realname is set in From:

This should avoid further 'Encoding could not be reliably detected'
errors for messages with 'From: addr <AT> dom.ain' headers.

 lib/utils.rb | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/utils.rb b/lib/utils.rb
index cc9fd46..e3a4836 100644
--- a/lib/utils.rb
+++ b/lib/utils.rb
@@ -61,7 +61,10 @@ module Ag
 
     def get_sender_displayname(message)
       begin
-        fix_encoding(message[:from].addrs.first.display_name).strip
+        display_name = message[:from].addrs.first.display_name
+        display_name ||= message[:from].addrs.first.to_s
+
+        fix_encoding(display_name).strip
       rescue NoMethodError
         fix_encoding(message[:from].to_s).strip
       end