email header wrapping and truncation

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

email header wrapping and truncation

Jack
Greetings all,

This is a problem that has bitten me in the past, but I finally tracked  
it down, with sufficient examples to demonstrate the problem.

In most cases, email headers (in the actual message) are

Header:  content

However, for some long values, it appears as

Header:
  content

If the content is long enough, you get

Header:
  first part of header, but tr
          uncated and wrapped.

It seems that in this last case, Balsa only sees the first line as the  
value of that header ("first part of header, but tr") which can cause a  
problem, if, for example, it is the "Reply-to:" header.  I now have two  
examples of this actually happening.  One is marketing messages I used  
to get from consumer cellular.  The other is from the gentoo.org  
mailing list system.  If I send a subscribe reqeust, it replies with a  
message I need to reply to to confirm the request.  However, it's  
Reply-to is of the third form above, and not noticing the truncation,  
my first reply just got me an error bounce.

Most interesting however, is that when I went to find the comsumer  
cellular example, I found a fourth type:

Header: Initial part o
  f header, with more here, and th
          e remainder here

In this case, Balsa gets the entire content of the header without  
problem.

I suppose the difference between the third and fourth types depends on  
one of the mail handlers between the sender and the one I get the  
message from by POP3.  However, I don't know if consumer cellular  
changed their system, or something changed someplace else in between,  
or perhaps some change in Balse (or a library) as all examples I can  
find of the third type are older than the fourth type.

Sorry for rambling, but hopefully this explains my issue enough for  
someone to either confirm it or point out what I'm doing wrong.

Let me know if you need any other info, such as versions of libraries  
my Balsa is compiled against.

Thanks for any help.

Jack
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: email header wrapping and truncation

Peter Bloomfield
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Jack,

On 07/21/2017 03:18:56 PM Fri, Jack wrote:

> Greetings all,
>
> This is a problem that has bitten me in the past, but I finally tracked it down, with sufficient examples to demonstrate the problem.
>
> In most cases, email headers (in the actual message) are
>
> Header:  content
>
> However, for some long values, it appears as
>
> Header:
>  content
>
> If the content is long enough, you get
>
> Header:
>  first part of header, but tr
>          uncated and wrapped.
>
> It seems that in this last case, Balsa only sees the first line as the value of that header ("first part of header, but tr") which can cause a problem, if, for example, it is the "Reply-to:" header.  I now have two examples of this actually happening.  One is marketing messages I used to get from consumer cellular.  The other is from the gentoo.org mailing list system.  If I send a subscribe reqeust, it replies with a message I need to reply to to confirm the request.  However, it's Reply-to is of the third form above, and not noticing the truncation, my first reply just got me an error bounce.
>
> Most interesting however, is that when I went to find the comsumer cellular example, I found a fourth type:
>
> Header: Initial part o
>  f header, with more here, and th
>          e remainder here
>
> In this case, Balsa gets the entire content of the header without problem.
>
> I suppose the difference between the third and fourth types depends on one of the mail handlers between the sender and the one I get the message from by POP3.  However, I don't know if consumer cellular changed their system, or something changed someplace else in between, or perhaps some change in Balse (or a library) as all examples I can find of the third type are older than the fourth type.
>
> Sorry for rambling, but hopefully this explains my issue enough for someone to either confirm it or point out what I'm doing wrong.
>
> Let me know if you need any other info, such as versions of libraries my Balsa is compiled against.
>
> Thanks for any help.
>
> Jack

Sorry to read about the problems you're seeing!

Those continuation lines (beginning with WSP: space or tab) are all supposed to be 'unfolded' by concatenating them with the 'Header:' line. Balsa hands off the whole job of parsing the message to GMime, which then provides unfolded headers on request. The GMime code that handles unfolding has been stable for many years.

Perhaps the lines that don't get concatenated have some other non-printing character before the WSP. Where do you see the folded headers? In an mbox? Can you check for other spurious characters?

Best,

Peter
-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQS030wPRfNNA5alz3MfX9S1uSp09QUCWXJthgAKCRAfX9S1uSp0
9TSuAJ9lmvonxEkT8Nk8TXTHSPmum/HiKwCfayw9JBJHu+Ddf12P3cTGSXzotoI=
=7P7v
-----END PGP SIGNATURE-----
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: email header wrapping and truncation

Jack
Hi Peter,

On 2017.07.21 17:09, Peter Bloomfield wrote:

> Hi Jack,
>
> On 07/21/2017 03:18:56 PM Fri, Jack wrote:
>> Greetings all,
>>
>> This is a problem that has bitten me in the past, but I finally  
>> tracked it down, with sufficient examples to demonstrate the problem.
>>
>> In most cases, email headers (in the actual message) are
>>
>> Header:  content
>>
>> However, for some long values, it appears as
>>
>> Header:
>>  content
>>
>> If the content is long enough, you get
>>
>> Header:
>>  first part of header, but tr
>>          uncated and wrapped.
>>
>> It seems that in this last case, Balsa only sees the first line as  
>> the value of that header ("first part of header, but tr") which can  
>> cause a problem, if, for example, it is the "Reply-to:" header.  I  
>> now have two examples of this actually happening.  One is marketing  
>> messages I used to get from consumer cellular.  The other is from  
>> the gentoo.org mailing list system.  If I send a subscribe reqeust,  
>> it replies with a message I need to reply to to confirm the  
>> request.  However, it's Reply-to is of the third form above, and not  
>> noticing the truncation, my first reply just got me an error bounce.
>>
>> Most interesting however, is that when I went to find the comsumer  
>> cellular example, I found a fourth type:
>>
>> Header: Initial part o
>>  f header, with more here, and th
>>          e remainder here
>>
>> In this case, Balsa gets the entire content of the header without  
>> problem.
>>
>> I suppose the difference between the third and fourth types depends  
>> on one of the mail handlers between the sender and the one I get the  
>> message from by POP3.  However, I don't know if consumer cellular  
>> changed their system, or something changed someplace else in  
>> between, or perhaps some change in Balse (or a library) as all  
>> examples I can find of the third type are older than the fourth type.
>>
>> Sorry for rambling, but hopefully this explains my issue enough for  
>> someone to either confirm it or point out what I'm doing wrong.
>>
>> Let me know if you need any other info, such as versions of  
>> libraries my Balsa is compiled against.
>>
>> Thanks for any help.
>>
>> Jack
>
> Sorry to read about the problems you're seeing!
>
> Those continuation lines (beginning with WSP: space or tab) are all  
> supposed to be 'unfolded' by concatenating them with the 'Header:'  
> line. Balsa hands off the whole job of parsing the message to GMime,  
> which then provides unfolded headers on request. The GMime code that  
> handles unfolding has been stable for many years.
>
> Perhaps the lines that don't get concatenated have some other  
> non-printing character before the WSP. Where do you see the folded  
> headers? In an mbox? Can you check for other spurious characters?
>
> Best,
>
> Peter
My inbox is an mbox, but I generally move almost everything to maildir  
folders, so that's where all my recent examples come from.  I haven't  
noticed any odd characters (just looking with grep or in emacs) so I'll  
have to dig back in and try some "od -c".  Since the only thing exposed  
would be email addresses, I suppose I can extract different examples  
into an attachment.

I've had gmime 2.6.23 installed since February, but had this problem  
with Consumer Cellular early 2016 and with Gentoo mailing lists just  
the past few days.  I suppose "Reply-to" is one of the few headers  
(other than "From" and "CC" which frequently get used, but only  
infrequently gets a really long value.  In the case of the mailing  
lists, it's not a "real" address, but is parsed by the list software so  
all it needs is that address and the subject to know what to do.

Jack
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: email header wrapping and truncation

Albrecht Dreß
In reply to this post by Jack
Hi Jack:

Sorry for the late reply, I've been on vacation and returned only yesterday…

Am 21.07.17 21:18 schrieb(en) Jack:
> It seems that in this last case, Balsa only sees the first line as the value of that header ("first part of header, but tr") which can cause a problem, if, for example, it is the "Reply-to:" header.  I now have two examples of this actually happening.  One is marketing messages I used to get from consumer cellular.  The other is from the gentoo.org mailing list system.  If I send a subscribe reqeust, it replies with a message I need to reply to to confirm the request.  However, it's Reply-to is of the third form above, and not noticing the truncation, my first reply just got me an error bounce.

How do you load the message from your ISP's server (POP3, IMAP, fetchmail, etc.)?  If you use Balsa's built-in POP3 interface to load them, it might be helpful to enable debugging to check if there is an issue with it (Peter discovered a broken POP3 server recently):

- terminate Balsa
- run Balsa by calling
   G_MESSAGES_DEBUG=all /path/to/balsa > LOG.out 2> LOG.err
- download the message
- terminate Balsa

You will find the download log by running “grep DEBUG LOG.out”, maybe this will give some more information.  If you want to post it here, please remember to anonymise or remove any sensitive information!

Cheers,
Albrecht.
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list

attachment0 (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: email header wrapping and truncation

Jack
On 2017.08.02 06:48, Albrecht Dreß wrote:
> Hi Jack:
>
> Sorry for the late reply, I've been on vacation and returned only  
> yesterday…
I hope you enjoyed yourself.  This is certainly not a critical issue.

> Am 21.07.17 21:18 schrieb(en) Jack:
>> It seems that in this last case, Balsa only sees the first line as  
>> the value of that header ("first part of header, but tr") which can  
>> cause a problem, if, for example, it is the "Reply-to:" header.  I  
>> now have two examples of this actually happening.  One is marketing  
>> messages I used to get from consumer cellular.  The other is from  
>> the gentoo.org mailing list system.  If I send a subscribe reqeust,  
>> it replies with a message I need to reply to to confirm the  
>> request.  However, it's Reply-to is of the third form above, and not  
>> noticing the truncation, my first reply just got me an error bounce.
>
> How do you load the message from your ISP's server (POP3, IMAP,  
> fetchmail, etc.)?  If you use Balsa's built-in POP3 interface to load  
> them, it might be helpful to enable debugging to check if there is an  
> issue with it (Peter discovered a broken POP3 server recently):
>
> - terminate Balsa
> - run Balsa by calling
>   G_MESSAGES_DEBUG=all /path/to/balsa > LOG.out 2> LOG.err
> - download the message
> - terminate Balsa
>
> You will find the download log by running “grep DEBUG LOG.out”, maybe  
> this will give some more information.  If you want to post it here,  
> please remember to anonymise or remove any sensitive information!

I do use POP3, with the added wrinkle of going through POPFILE for spam  
filtering.  I will try this logging, but I also still need to do a  
"complete" test of sending myself a test with long headers, and  
capturing the file at each stage: unsent from draft folder, from sent  
folder, and then as received.  (also using different send and receive  
hosts)  The little testing I have done just left me more confused.  
Unfortunately, since I am now aware of it, as long as I can remember to  
check the TO: header before actually sending any reply, I can work  
around it.

Jack
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Loading...