[gentoo-user] recompiling vim linked to libncursesw

[gentoo-user] recompiling vim linked to libncursesw

[Fernando Canizo]

Hi all.

I'm having trouble with my encoding using mutt + vim + utf-8,
basically mi emails are sent with wrong encoding when *replying*. I've
tracked the problem, searched, readed FAQs and i found that maybe my
problem is this: that while mutt is linked to libncursesw (wide
library) vim is to libncurses (normal), this is the output of ldd:

$ ldd `which vim` | grep curses
libncurses.so.5 => /lib/libncurses.so.5 (0xb7f98000)
$ ldd `which mutt` | grep curses
libncursesw.so.5 => /lib/libncursesw.so.5 (0xb7f8f000)

Except for this my utf-8 environment is running fine.

I tried re 'emerge vim' thinking that maybe it just get fixed since i
use unicode and utf8 in my USE, but that didn't do anything, vim is
still linked to the non wide curses library.

So the question is: how do i tell to emerge to forge a vim wich is
linked to libncursesw.so ?

-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
snappy repartee:
	What you'd say if you had another chance.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] recompiling vim linked to libncursesw

[Richard Fish]

Fernando Canizo wrote:

>Hi all.
>
>I'm having trouble with my encoding using mutt + vim + utf-8,
>basically mi emails are sent with wrong encoding when *replying*. I've
>tracked the problem, searched, readed FAQs and i found that maybe my
>problem is this: that while mutt is linked to libncursesw (wide
>library) vim is to libncurses (normal), this is the output of ldd:
>
>$ ldd `which vim` | grep curses
>libncurses.so.5 => /lib/libncurses.so.5 (0xb7f98000)
>$ ldd `which mutt` | grep curses
>libncursesw.so.5 => /lib/libncursesw.so.5 (0xb7f8f000)
>
>Except for this my utf-8 environment is running fine.
>
>I tried re 'emerge vim' thinking that maybe it just get fixed since i
>use unicode and utf8 in my USE, but that didn't do anything, vim is
>still linked to the non wide curses library.
>
>So the question is: how do i tell to emerge to forge a vim wich is
>linked to libncursesw.so ?
>  
>

Hmm, tough one.  I don't see any way to configure vim specifically for 
ncursesw.  Your best bet may be to try and fake out the dynamic linker 
and tell it to use libncursesw instead of libncurses for vim.

ln -s /lib/libncursesw.so.5 /etc/vim/libncurses.so.5
LD_LIBRARY_PATH=/etc/vim ldd /usr/bin/vim
LD_LIBRARY_PATH=/etc/vim vim

If vim seems sane, then exporting LD_LIBRARY_PATH before starting mutt 
should resolve the problem (assuming that the problem really is in the 
type of ncurses...)

-Richard



-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: recompiling vim linked to libncursesw

[Moshe Kaminsky]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

* Fernando Canizo <conan@lugmen.org.ar> [27/07/05 07:15]:
> 
> Hi all.
> 
> I'm having trouble with my encoding using mutt + vim + utf-8,
> basically mi emails are sent with wrong encoding when *replying*. I've
> tracked the problem, searched, readed FAQs and i found that maybe my
> problem is this: that while mutt is linked to libncursesw (wide
> library) vim is to libncurses (normal), this is the output of ldd:

I find it hard to believe that this is the problem. You say that you can 
use utf8 when you are composing (or writing some other stuff), right? 
What are the values of 'encoding' and 'fileencoding' in vim when 
replying?

Moshe

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] recompiling vim linked to libncursesw

[Philip Webb]

050727 Richard Fish wrote:
> Fernando Canizo wrote:
>> $ ldd `which vim` | grep curses
>> libncurses.so.5 => /lib/libncurses.so.5 (0xb7f98000)
>> $ ldd `which mutt` | grep curses
>> libncursesw.so.5 => /lib/libncursesw.so.5 (0xb7f8f000)

I too use Mutt + Vim -- ie Gvim -- & sometimes Vim + UTF8 (not with Mutt)
& I never have problems.  However, my Mutt doesn't seem to use Curses :

  root: log> ldd `which vim` | grep curses
          libncurses.so.5 => /lib/libncurses.so.5 (0xb7f9a000)
  root: log> ldd `which mutt` | grep curses
  root: log> emerge -pv mutt 
  ...
  [ebuild R ] mail-client/mutt-1.5.8-r2 
   -buffysize -cjk +crypt -imap -mbox -nls -nntp -sasl +slang +ssl -vanilla

-- 
========================,,============================================
SUPPORT     ___________//___,  Philip Webb : purslow@chass.utoronto.ca
ELECTRIC   /] [] [] [] [] []|  Centre for Urban & Community Studies
TRANSIT    `-O----------O---'  University of Toronto
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] recompiling vim linked to libncursesw

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 790 bytes --]

On Wed, Jul 27, 2005 at 01:12:23AM -0300, Fernando Canizo wrote:
> Hi all.
> 
> I'm having trouble with my encoding using mutt + vim + utf-8,
> basically mi emails are sent with wrong encoding when *replying*. I've
> tracked the problem, searched, readed FAQs and i found that maybe my
> problem is this: that while mutt is linked to libncursesw (wide
> library) vim is to libncurses (normal), this is the output of ldd:

It won't matter for vim.  vim is a termcap application, and is doing the
interpretation of locales by itself.  The distinction between ncurses and
ncursesw only applies to curses applications.  (ncurses provides low-level
interfaces for termcap and terminfo applications).

-- 
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-user] recompiling vim linked to libncursesw

[Fernando Canizo]

El Wed, Jul 27, 2005 at 07:08:27AM +0200, Richard Fish me decía:
> Fernando Canizo wrote:
> >I'm having trouble with my encoding using mutt + vim + utf-8,
> > [snip]
> Hmm, tough one.  I don't see any way to configure vim specifically for 
> ncursesw.  Your best bet may be to try and fake out the dynamic linker 
> and tell it to use libncursesw instead of libncurses for vim.
> 
> ln -s /lib/libncursesw.so.5 /etc/vim/libncurses.so.5
> LD_LIBRARY_PATH=/etc/vim ldd /usr/bin/vim
> LD_LIBRARY_PATH=/etc/vim vim
> 
> If vim seems sane, then exporting LD_LIBRARY_PATH before starting mutt 
> should resolve the problem (assuming that the problem really is in the 
> type of ncurses...)

Thanks Richard, but didn't work. And now i'm not so sure that this
could ve the problem, after all i can create some file without problem
in vim, put some accents and view ok after, and file says it's utf-8.

I'll gonna rephrase my problem giving more info.

-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
Maybe I should just cut my losses, give up on Lisa, and make a fresh
start with Maggie.

		-- Homer Simpson
		   Lisa's Pony
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] recompiling vim linked to libncursesw

[Fernando Canizo]

El Wed, Jul 27, 2005 at 04:38:42AM -0400, Philip Webb me decía:
> 050727 Richard Fish wrote:
> > Fernando Canizo wrote:
> >> $ ldd `which vim` | grep curses
> >> libncurses.so.5 => /lib/libncurses.so.5 (0xb7f98000)
> >> $ ldd `which mutt` | grep curses
> >> libncursesw.so.5 => /lib/libncursesw.so.5 (0xb7f8f000)
> 
> I too use Mutt + Vim -- ie Gvim -- & sometimes Vim + UTF8 (not with Mutt)
> & I never have problems.  However, my Mutt doesn't seem to use Curses :
> 
>   root: log> ldd `which vim` | grep curses
>           libncurses.so.5 => /lib/libncurses.so.5 (0xb7f9a000)
>   root: log> ldd `which mutt` | grep curses
>   root: log> emerge -pv mutt 
>   ...
>   [ebuild R ] mail-client/mutt-1.5.8-r2 
>    -buffysize -cjk +crypt -imap -mbox -nls -nntp -sasl +slang +ssl -vanilla

Yes, i have it that way before, and the same problem. But then i see
the mutt FAQ (http://wiki.mutt.org/index.cgi?MuttFaq/Charset) which says:

"Default Slang seems not to work with UTF-8, relink Mutt against
libncursesw. (Hello Gentoo users :-)"

So, maybe this is an old issue, but the first step i choose to do was
to relink to ncurses and give up slang, that way i don't have to keep
an eye on versions and stuff, in fact only mutt was using slang on my
system, so nothing is missed.

Besides, the problem already was there when using slang.

-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
panic ("Splunge!");
        linux-2.2.16/drivers/scsi/psi240i.c
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: recompiling vim linked to libncursesw

[Fernando Canizo]

El Wed, Jul 27, 2005 at 09:04:29AM +0300, Moshe Kaminsky me decía:
> * Fernando Canizo <conan@lugmen.org.ar> [27/07/05 07:15]:
> > 
> > Hi all.
> > 
> > I'm having trouble with my encoding using mutt + vim + utf-8,
> > basically mi emails are sent with wrong encoding when *replying*. I've
> > tracked the problem, searched, readed FAQs and i found that maybe my
> > problem is this: that while mutt is linked to libncursesw (wide
> > library) vim is to libncurses (normal), this is the output of ldd:
> 
> I find it hard to believe that this is the problem. You say that you can 
> use utf8 when you are composing (or writing some other stuff), right? 
> What are the values of 'encoding' and 'fileencoding' in vim when 
> replying?
> Moshe

Like i said to Richard, maybe you're right. I mean: i can write an
utf-8 file from scratch using vim alone, so why would not when
invoking vim from mutt? Maybe is that mutt is telling vim something
incorrect when they communicate.

Well, i'll give more information, but this gonna grow large ;)

Reading the mutt FAQ (http://wiki.mutt.org/index.cgi?MuttFaq/Charset)
and checking everything is ok:

locale seems to be ok:
~$ locale
LANG=es_AR.utf-8
LC_CTYPE="es_AR.utf-8"
LC_NUMERIC="es_AR.utf-8"
LC_TIME="es_AR.utf-8"
LC_COLLATE="es_AR.utf-8"
LC_MONETARY="es_AR.utf-8"
LC_MESSAGES="es_AR.utf-8"
LC_PAPER="es_AR.utf-8"
LC_NAME="es_AR.utf-8"
LC_ADDRESS="es_AR.utf-8"
LC_TELEPHONE="es_AR.utf-8"
LC_MEASUREMENT="es_AR.utf-8"
LC_IDENTIFICATION="es_AR.utf-8"
LC_ALL=es_AR.utf-8

the locales settings are supported:
~$ locale -a
C
es_AR
es_AR.utf8
POSIX

checking if the locales work with perl:
~$ perl -e ""
ok doesn't show anything

checking if perl is doing the right things by setting an erroneous
locale:
~$ env LC_ALL=nocharset perl -e ""
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "es_AR.utf-8",
	LC_ALL = "nocharset",
	LC_CTYPE = "es_AR.utf-8",
	LANG = "es_AR.utf-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

ok, it cries, so it's working ok

my ~/.signature is in utf-8, my ~/.alias is too

i got this in my ~/.vimrc:
set encoding=utf-8
set fileencoding=utf-8
set termencoding=utf-8

and when mutt invokes vim i re-check that this is ok, and is ok (i
mean i check in runtime and it obbeys the configuration)

i got this in my ~/.muttrc:
set send_charset="us-ascii:utf-8"
set charset="utf-8"
set locale="es_AR.utf8"

from the mutt man i know this settings should not be necessary, since
the system is configured ok, but i try with and without and get no
difference.

Ok, that's all concerning configuration. Now i tell you how the
problem works: in mutt, if i compose a mail from scratch, without
anything, not even signature, and put a LATIN SMALL LETTER A WITH
ACUTE (got that name from unicode chart), and then send it to myself,
and to a friend, my friend sees it ok and i too.

But if now i reply to this same mail, when vim comes with the quoted
text that mutt passes to it y see garbage.

So mutt is ok seeing and sending utf-8, vim is ok writing and reading
utf-8, but when both "cooperate", things get screwed.

I investigate what was in the archives, so i saved a copy (using 'C'
command from mutt) of the first message (the one i receive from me)
and file says: 'UTF-8 Unicode mail text', check what's inside with
hexedit and see that LATIN SMALL LETTER A WITH ACUTE is encoded with
this hex: C3 A1 (which is not 00 E1 from unicode chart from
http://www.unicode.org/charts/)

Then i got that same mail and press 'r' from mutt to respond, comes
vim with garbled text, and without touching anything i save it under
some other name, and then cancel message, file in this saved text
gives me: 'UTF-8 Unicode text', but when i see inside with hexedit, i
got this hex for the same letter: C3 83 C2 A1, so now i have 4 bytes
instead of the too before. So vim-mutt (?) is re-encoding the stuff.

Like i said before, checking in vim when called from mutt for enc, and
fenc gives utf-8 like it should.

I create a file with vim to check this differences with the unicode
chart and i got C3 A1 too, so maybe the problem is with vim, it should
put 00 E1 for LATIN SMALL LETTER A WITH ACUTE.

Well, that's all i remember to have done for this problem. I'm with
this for 4 months now, i think, i took the problem, get tired of
searching and testing stuff, leave for a month, get to charge again,
and so on. But now i really want to solve it.

I think i'm going to crosspost this to vim and mutt mailing lists. But
if someone here knows how to solve this, i appreciate any help, tip,
direction to look or search, or maybe praying would do the job. 

Linking after build vim to libncursesw the way Richard say a couple of
mails before didn't solve it. If there is a way to say to emerge that
link vim to this library from the beginning i would like to try it.

Besides if anyone knows how to get which programs are using a library
(i have done it before, but at this time my brain is screwed).

What i didn't tried yet is 'the redmond way', i want to stay away from
that metod, if possible.

Thanks in advance.

-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
Antonym, n.:
	The opposite of the word you're trying to think of.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] recompiling vim linked to libncursesw

[Fernando Canizo]

El Wed, Jul 27, 2005 at 05:59:53AM -0400, Thomas Dickey me decía:
> On Wed, Jul 27, 2005 at 01:12:23AM -0300, Fernando Canizo wrote:
> > Hi all.
> > 
> > I'm having trouble with my encoding using mutt + vim + utf-8,
> > basically mi emails are sent with wrong encoding when *replying*. I've
> > tracked the problem, searched, readed FAQs and i found that maybe my
> > problem is this: that while mutt is linked to libncursesw (wide
> > library) vim is to libncurses (normal), this is the output of ldd:
> 
> It won't matter for vim.  vim is a termcap application, and is doing the
> interpretation of locales by itself.  The distinction between ncurses and
> ncursesw only applies to curses applications.  (ncurses provides low-level
> interfaces for termcap and terminfo applications).

But vim is linked against libncurses, doesn't make it a 'curses
application' ?

-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
QOTD:
	"There may be no excuse for laziness, but I'm sure looking."
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: recompiling vim linked to libncursesw

[Moshe Kaminsky]

[-- Attachment #1: Type: text/plain, Size: 3980 bytes --]

Hi,
* Fernando Canizo <conan@lugmen.org.ar> [27/07/05 14:14]:
> 
> El Wed, Jul 27, 2005 at 09:04:29AM +0300, Moshe Kaminsky me decía:
> > * Fernando Canizo <conan@lugmen.org.ar> [27/07/05 07:15]:
> > > 
> > > Hi all.
> > > 
> > > I'm having trouble with my encoding using mutt + vim + utf-8,
> > > basically mi emails are sent with wrong encoding when *replying*. I've
> > > tracked the problem, searched, readed FAQs and i found that maybe my
> > > problem is this: that while mutt is linked to libncursesw (wide
> > > library) vim is to libncurses (normal), this is the output of ldd:
> > 
> > I find it hard to believe that this is the problem. You say that you can 
> > use utf8 when you are composing (or writing some other stuff), right? 
> > What are the values of 'encoding' and 'fileencoding' in vim when 
> > replying?
> > Moshe
> 
> Like i said to Richard, maybe you're right. I mean: i can write an
> utf-8 file from scratch using vim alone, so why would not when
> invoking vim from mutt? Maybe is that mutt is telling vim something
> incorrect when they communicate.
> 
> Well, i'll give more information, but this gonna grow large ;)
> 
[snip]
> 
> checking if the locales work with perl:
> ~$ perl -e ""
> ok doesn't show anything

That's one use of perl I never thought about :)
> 
> checking if perl is doing the right things by setting an erroneous
> locale:
> ~$ env LC_ALL=nocharset perl -e ""
> perl: warning: Setting locale failed.
> perl: warning: Please check that your locale settings:
> 	LANGUAGE = "es_AR.utf-8",
> 	LC_ALL = "nocharset",
> 	LC_CTYPE = "es_AR.utf-8",
> 	LANG = "es_AR.utf-8"
>     are supported and installed on your system.
> perl: warning: Falling back to the standard locale ("C").
> 
> ok, it cries, so it's working ok
> 
> my ~/.signature is in utf-8, my ~/.alias is too
> 
> i got this in my ~/.vimrc:
> set encoding=utf-8
> set fileencoding=utf-8

Please try removing this setting, then check the value after vim reads 
the file (when you reply). Vim sets this option when editing an existing 
file according to what it thinks the encoding of the file to be. Also, 
you might want to try something like

:e ++enc=utf-8 <file>

This will force vim to read this file as a utf-8 file. Also, what is the 
value of 'fileencodings'?

> set termencoding=utf-8
> 
> and when mutt invokes vim i re-check that this is ok, and is ok (i
> mean i check in runtime and it obbeys the configuration)
> 
> i got this in my ~/.muttrc:
> set send_charset="us-ascii:utf-8"

Might want to try just "utf-8", but I don't think it will matter.

> set charset="utf-8"
> set locale="es_AR.utf8"
> 
> from the mutt man i know this settings should not be necessary, since
> the system is configured ok, but i try with and without and get no
> difference.
> 
> Ok, that's all concerning configuration. Now i tell you how the
> problem works: in mutt, if i compose a mail from scratch, without
> anything, not even signature, and put a LATIN SMALL LETTER A WITH
> ACUTE (got that name from unicode chart), and then send it to myself,
> and to a friend, my friend sees it ok and i too.
> 
> But if now i reply to this same mail, when vim comes with the quoted
> text that mutt passes to it y see garbage.

Can you include one of this characters in your reply?

> 
> So mutt is ok seeing and sending utf-8, vim is ok writing and reading
> utf-8, but when both "cooperate", things get screwed.
> 
> I investigate what was in the archives, so i saved a copy (using 'C'
> command from mutt) of the first message (the one i receive from me)
> and file says: 'UTF-8 Unicode mail text', check what's inside with
> hexedit and see that LATIN SMALL LETTER A WITH ACUTE is encoded with
> this hex: C3 A1 (which is not 00 E1 from unicode chart from
> http://www.unicode.org/charts/)

I think this is just the way these characters are represented in utf-8.

Moshe

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] recompiling vim linked to libncursesw

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]

On Wed, Jul 27, 2005 at 08:11:05AM -0300, Fernando Canizo wrote:
> But vim is linked against libncurses, doesn't make it a 'curses
> application' ?

no.  A curses application calls initscr() or newterm(), uses the curses
library to perform optimized display updates.  A termcap application
calls tgetent(), and uses tputs() or tgoto() to write strings directly
to the display.

Since vim is a termcap application, you could in principle link it with
libtermcap without seeing any difference in vim's functionality.

There are some applications that can be configured (compiled) for any of
these interfaces:

	termcap
	terminfo
	curses (ncurses)
	ncursesw

tin and vile do this, for example.

-- 
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-user] Re: recompiling vim linked to libncursesw

[Andreas Claesson]

On 7/27/05, Moshe Kaminsky <kaminsky@math.huji.ac.il> wrote:
> Hi,
> * Fernando Canizo <conan@lugmen.org.ar> [27/07/05 14:14]:
<snip>
> > I investigate what was in the archives, so i saved a copy (using 'C'
> > command from mutt) of the first message (the one i receive from me)
> > and file says: 'UTF-8 Unicode mail text', check what's inside with
> > hexedit and see that LATIN SMALL LETTER A WITH ACUTE is encoded with
> > this hex: C3 A1 (which is not 00 E1 from unicode chart from
> > http://www.unicode.org/charts/)
> 
> I think this is just the way these characters are represented in utf-8.

Yes, it is.

00E1 hex is '0000000 11100001' in binary.

When encoding this as UTF-8 this value is stored in two bytes.

The last byte will begin with '10' followed by the last 6 bits of data. 

'10 100001' binary or 'A1' in hex.

The first byte will begin with '110' to indicate that it is a two byte 
character followed by the remaining significant data. 

'110 00011' binary or 'C3' hex.

This is correct.

The problem seem to be that mutt(?) takes this UTF-8 encoded data
and encodes as UTF-8 again as if the data was two 8 bit characters.
 
'C3' then becomes 'C3 83' and 'A1' becomes 'C2 A1' 


/Andreas

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: recompiling vim linked to libncursesw

[Matan Peled]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Fernando Canizo wrote:
> Yes!
> In the next line a LATIN SMALL LETTER A WITH ACUTE:
> �
This doesn't seem to be UTF-8 (I get a box in UTF-8 mode,)
but it looks OK in latin-1.
> But you should see that ok, now if i reply to that message, when it
> reaches me through the list, i will send this stuff:
> á
Now this looks like UTF-8.
> which seems to be (not sure about this) the codification of the two
> bytes of the LATIN SMALL LETTER A WITH ACUTE but un latin1.

- --
[Name      ]   ::  [Matan I. Peled    ]
[Location  ]   ::  [Israel            ]
[Public Key]   ::  [0xD6F42CA5        ]
[Keyserver ]   ::  [keyserver.kjsl.com]
encrypted/signed  plain text  preferred

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFC54MGA7Qvptb0LKURAknKAJ9a97ZLS6r16JmNYUjauD5yoepWWgCePt8g
cPcyu0FqLM9xtTO5eUZU8sU=
=guht
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: recompiling vim linked to libncursesw

[Fernando Canizo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 3097 bytes --]

El Wed, Jul 27, 2005 at 02:45:27PM +0300, Moshe Kaminsky me decía:
> > i got this in my ~/.vimrc:
> > set encoding=utf-8
> > set fileencoding=utf-8
> 
> Please try removing this setting, then check the value after vim reads 
> the file (when you reply). Vim sets this option when editing an existing 
> file according to what it thinks the encoding of the file to be. Also, 
> you might want to try something like

A-jhá! I remove those settings from mi ~/.vimrc and when replying to a
mail that contains utf-8 (some mail that i sent myself) i get:
encoding=utf-8
but:
fileencoding=latin1

So, mutt is showing correctly an utf-8 mail, but when it creates the
quoted mail for reply in /tmp is giving vim a recoded file.

I think... is that? So, what's happening here?

> :e ++enc=utf-8 <file>

don't know how to use this from the call from mutt

> This will force vim to read this file as a utf-8 file. Also, what is the 
> value of 'fileencodings'?

fileencodings=ucs-bom,utf-8,latin1,default

i already have tried with
fileencodings=utf-8
only and the problem stays.

> > i got this in my ~/.muttrc:
> > set send_charset="us-ascii:utf-8"
> 
> Might want to try just "utf-8", but I don't think it will matter.

Oh, no, i already tried it, i put before:
set send_charset="utf-8"
set send_charset="us-ascii:iso-8859-1:utf-8"
and the problem stays too.

> > Ok, that's all concerning configuration. Now i tell you how the
> > problem works: in mutt, if i compose a mail from scratch, without
> > anything, not even signature, and put a LATIN SMALL LETTER A WITH
> > ACUTE (got that name from unicode chart), and then send it to myself,
> > and to a friend, my friend sees it ok and i too.
> > 
> > But if now i reply to this same mail, when vim comes with the quoted
> > text that mutt passes to it y see garbage.
> 
> Can you include one of this characters in your reply?

Yes!
In the next line a LATIN SMALL LETTER A WITH ACUTE:
á
But you should see that ok, now if i reply to that message, when it
reaches me through the list, i will send this stuff:
á
which seems to be (not sure about this) the codification of the two
bytes of the LATIN SMALL LETTER A WITH ACUTE but un latin1.

Why is that? I don't know.

> > I investigate what was in the archives, so i saved a copy (using 'C'
> > command from mutt) of the first message (the one i receive from me)
> > and file says: 'UTF-8 Unicode mail text', check what's inside with
> > hexedit and see that LATIN SMALL LETTER A WITH ACUTE is encoded with
> > this hex: C3 A1 (which is not 00 E1 from unicode chart from
> > http://www.unicode.org/charts/)
> 
> I think this is just the way these characters are represented in utf-8.

Yes it is. I were looking at the worng charts, the same was said to me in
the vim list.
-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
Jacquin's Postulate on Democratic Government:
	No man's life, liberty, or property are safe while the
	legislature is in session.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: recompiling vim linked to libncursesw

[Fernando Canizo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 656 bytes --]

El Wed, Jul 27, 2005 at 02:39:58PM +0200, Andreas Claesson me decía:
> The problem seem to be that mutt(?) takes this UTF-8 encoded data
> and encodes as UTF-8 again as if the data was two 8 bit characters.
>  
> 'C3' then becomes 'C3 83' and 'A1' becomes 'C2 A1' 

Yes!, yes! something like that is happening (maybe is re-encoding in
latin1, but you seem to be the master of encodings, so sure i'm wrong
with this), but i don't know what to do about.

-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
See you in hell, candy boys!!

		-- Homer Simpson
		   Homer Badman
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] recompiling vim linked to libncursesw

[Fernando Canizo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 1474 bytes --]

El Wed, Jul 27, 2005 at 08:27:12AM -0400, Thomas Dickey me decía:
> On Wed, Jul 27, 2005 at 08:11:05AM -0300, Fernando Canizo wrote:
> > But vim is linked against libncurses, doesn't make it a 'curses
> > application' ?
> 
> no.  A curses application calls initscr() or newterm(), uses the curses
> library to perform optimized display updates.  A termcap application
> calls tgetent(), and uses tputs() or tgoto() to write strings directly
> to the display.
> 
> Since vim is a termcap application, you could in principle link it with
> libtermcap without seeing any difference in vim's functionality.
> 
> There are some applications that can be configured (compiled) for any of
> these interfaces:
> 
> 	termcap
> 	terminfo
> 	curses (ncurses)
> 	ncursesw
> 
> tin and vile do this, for example.

Cool, so i can discard that the problem comes from vim + libncurses.

In other mails from the thread we've reached to the point of knowing
that mutt is reencoding the chars. Since you seem to know about
libraries and calls to functions, do you know a method to track this
issue? I've been thinking maybe an strace to mutt, but y don't know
what to search for, and strace gives lots of info.

So the question is: what would you do to track this problem?

-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
woman: "Say something dirty to me!"
man: "Kitchen."
	-- meyerm
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: recompiling vim linked to libncursesw

[Fernando Canizo]

El Wed, Jul 27, 2005 at 03:50:14PM +0300, Matan Peled me decía:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Fernando Canizo wrote:
> > Yes!
> > In the next line a LATIN SMALL LETTER A WITH ACUTE:
> > �
> This doesn't seem to be UTF-8 (I get a box in UTF-8 mode,)
> but it looks OK in latin-1.
> > But you should see that ok, now if i reply to that message, when it
> > reaches me through the list, i will send this stuff:
> > á
> Now this looks like UTF-8.
> > which seems to be (not sure about this) the codification of the two
> > bytes of the LATIN SMALL LETTER A WITH ACUTE but un latin1.

Yes, i did saw the same. I forgot i was replying and forgot to check
fenc and enc in vim.

Now i will force the stuff:
setting fenc and enc to utf-8
...
Now i write a LATIN SMALL LETTER A WITH ACUTE:
á

-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
I had no shoes and I pitied myself.  Then I met a man who had no feet,
so I took his shoes.
		-- Dave Barry
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] recompiling vim linked to libncursesw

[Thomas Dickey]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

On Wed, Jul 27, 2005 at 10:24:23AM -0300, Fernando Canizo wrote:
> Cool, so i can discard that the problem comes from vim + libncurses.
> 
> In other mails from the thread we've reached to the point of knowing
> that mutt is reencoding the chars. Since you seem to know about
> libraries and calls to functions, do you know a method to track this
> issue? I've been thinking maybe an strace to mutt, but y don't know
> what to search for, and strace gives lots of info.

strace gives lots of information, but sometimes you can reduce it by
looking only at specific calls.  Usually what I'm interested in from
strace in this type of problem is the exec's and open's it shows.

I'm not sure what mutt's doing either.  But if it's recoding the information,
it's doing either via runtime library or via an external program.

Another tool is ltrace, which would (like strace, a lot of information)
show the library calls mutt makes.  That's useful in this sense to see
if it's doing something with iconv.
 
> So the question is: what would you do to track this problem?

I'd probably poke at the strace and ltrace results enough to get a sense
of generally what it's doing, then grep through mutt's sources to see
where it's doing _that_.

-- 
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[-- Attachment #2: Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-user] recompiling vim linked to libncursesw: solved!

[Fernando Canizo]

El Wed, Jul 27, 2005 at 10:10:34AM -0400, Thomas Dickey me decía:
> On Wed, Jul 27, 2005 at 10:24:23AM -0300, Fernando Canizo wrote:
> > Cool, so i can discard that the problem comes from vim + libncurses.
> >
> > In other mails from the thread we've reached to the point of knowing
> > that mutt is reencoding the chars. Since you seem to know about
> > libraries and calls to functions, do you know a method to track this
> > issue? I've been thinking maybe an strace to mutt, but y don't know
> > what to search for, and strace gives lots of info.
>
> strace gives lots of information, but sometimes you can reduce it by
> looking only at specific calls.  Usually what I'm interested in from
> strace in this type of problem is the exec's and open's it shows.
>
> I'm not sure what mutt's doing either.  But if it's recoding the information,
> it's doing either via runtime library or via an external program.
>
> Another tool is ltrace, which would (like strace, a lot of information)
> show the library calls mutt makes.  That's useful in this sense to see
> if it's doing something with iconv.
>
> > So the question is: what would you do to track this problem?
>
> I'd probably poke at the strace and ltrace results enough to get a sense
> of generally what it's doing, then grep through mutt's sources to see
> where it's doing _that_.

Well, thank you very much, and everybody else who has replied. Finally
i didn't need do poke through sources. The problem was that since i
change to utf-8 from latin1, my configuration files remained in
latin1, one of them was ~/.muttrc wich contained the first line of
this mail, with the word "decía" encoded in latin1.

So when mutt was compossing the temporal mail in /tmp for vim it found
"decía" and said to itself: "this is latin1, let's encode the rest
with this" and so it re-encoded my already utf-8 chars in utf-8 again.

The solution was to encode ~/.muttrc in utf-8. Also the same effect is
produced with an .alias or .signature file in another encoding, so be
aware! :)

Like i said, i've been through the FAQ, and searched the web using
google, the last thing after the mail from Thomas was narrow my search
using google groups and this string: "utf-8 vim mutt" and it gives 199
hits, from which i've read 50 when the reply come through mutt list. I
think i'm going to edith the mutt faq now since it's in a wiki.

Thank you everybody, i'm in testing mode, but i think this is it, it's
solved.
-- 
Fernando Canizo - LUGMen: www.lugmen.org.ar - A8N: a8n.lugmen.org.ar
The documentation is in Japanese.  Good luck.
	-- Rich $alz
-- 
gentoo-user@gentoo.org mailing list