Upcoming GMime 3.0 changes

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Upcoming GMime 3.0 changes

Gnome - Balsa mailing list
Hello Balsa devs,

I'm sending this email to inform you guys of some upcoming GMime 3.0 features that I've either implemented or will be adding soon.

1. Probably the most import change coming to GMime 3.0 is that each header field/value pair has its own raw value cache instead of using a cached stream of the entire head block, thus forcing the complete invalidation of the header cache when anything changed in the headers. In GMime 2.6, once the header cache was invalidated, each header would get re-folded when written back out to disk. This was obviously not ideal.

2. Next up is the replacement of the old custom GnuPG back-end with GpgMe. Also included with this change is full support for S/MIME via both multipart/signed and application/pkcs7-mime content-types using gpgsm (via GpgMe). During this change, I also took the liberty of simplifying the crypto API's a bit and so I was able to make it such that g_mime_multipart_signed_verify(), for example, no longer requires you to pass it a GMimeCryptoContext. Instead, GMime parses the Content-Type's protocol parameter and uses that to instantiate the correct crypto context (GMimeGpgContext for PGP and GMimePkcs7Context for S/MIME). Many of the various state properties have been replaced with bitflags that can be passed to encrypt() and decrypt(). The sign() method now also takes a detach argument (might make this into a bitflag instead?) in order to support encapsulated signing.

3. The GMimeMessage API's have been improved a bit. Gone are the old GMimeMessage API's that return strings for the From and Reply-To headers. Instead, *all* GMimeMessage API's return an InternetAddressList pointer which can be used to manipulate the parsed address header. Also improved is the g_mime_message_set_subject() method which now takes an optional charset parameter (or NULL to use the default).

4. New in GMime 3.0 is the GMimeParserOptions struct which can be passed to GMimeParser and other parser functions exposed in the lower-level API's. This structure helps define how strict/forgiving the various parsing routines should be with the input. This replaces the need for g_mime_init()'s flags so you can change these settings on the fly now.

5. Another new feature is GMimeTextPart which makes getting/setting text on a GMimePart simpler. GMimeTextPart is a subclass of GMimePart, so all of the GMimePart API's that you are used to still work but now you also have access to convenient functions like g_mime_text_part_get_text() and g_mime_text_part_set_text() which removes the need for creating streams, data wrappers, and setting them on the part.

6. Brand new rfc822 address parser which is more tolerant than the previous generation parser. What's not to love?

7. And finally we get to a nifty feature that I just hacked up while waiting for some other code to compile (hey, it takes an hour to compile... I needed something to do!) which is that GMimeParser now scans for -----BEGIN PGP MESSAGE-----/-----END PGP MESSAGE----- and -----BEGIN PGP SIGNED MESSAGE-----/-----END PGP SIGNED MESSAGE----- markers while looking for MIME boundaries and sets some state on the corresponding GMimePart that you can use to quickly decide if the part contains encapsulated OpenPGP data.

Jeff

_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Upcoming GMime 3.0 changes

Gnome - Balsa mailing list
Aaaaaand I've implemented support for inline PGP this morning 😊

Jeff

> -----Original Message-----
> From: balsa-list [mailto:[hidden email]] On Behalf Of Jeffrey
> Stedfast via balsa-list
> Sent: Monday, March 13, 2017 6:39 PM
> To: [hidden email]
> Subject: Upcoming GMime 3.0 changes
>
> Hello Balsa devs,
>
> I'm sending this email to inform you guys of some upcoming GMime 3.0
> features that I've either implemented or will be adding soon.
>
> 1. Probably the most import change coming to GMime 3.0 is that each header
> field/value pair has its own raw value cache instead of using a cached stream
> of the entire head block, thus forcing the complete invalidation of the header
> cache when anything changed in the headers. In GMime 2.6, once the
> header cache was invalidated, each header would get re-folded when
> written back out to disk. This was obviously not ideal.
>
> 2. Next up is the replacement of the old custom GnuPG back-end with
> GpgMe. Also included with this change is full support for S/MIME via both
> multipart/signed and application/pkcs7-mime content-types using gpgsm (via
> GpgMe). During this change, I also took the liberty of simplifying the crypto
> API's a bit and so I was able to make it such that
> g_mime_multipart_signed_verify(), for example, no longer requires you to
> pass it a GMimeCryptoContext. Instead, GMime parses the Content-Type's
> protocol parameter and uses that to instantiate the correct crypto context
> (GMimeGpgContext for PGP and GMimePkcs7Context for S/MIME). Many of
> the various state properties have been replaced with bitflags that can be
> passed to encrypt() and decrypt(). The sign() method now also takes a
> detach argument (might make this into a bitflag instead?) in order to support
> encapsulated signing.
>
> 3. The GMimeMessage API's have been improved a bit. Gone are the old
> GMimeMessage API's that return strings for the From and Reply-To headers.
> Instead, *all* GMimeMessage API's return an InternetAddressList pointer
> which can be used to manipulate the parsed address header. Also improved
> is the g_mime_message_set_subject() method which now takes an optional
> charset parameter (or NULL to use the default).
>
> 4. New in GMime 3.0 is the GMimeParserOptions struct which can be passed
> to GMimeParser and other parser functions exposed in the lower-level API's.
> This structure helps define how strict/forgiving the various parsing routines
> should be with the input. This replaces the need for g_mime_init()'s flags so
> you can change these settings on the fly now.
>
> 5. Another new feature is GMimeTextPart which makes getting/setting text
> on a GMimePart simpler. GMimeTextPart is a subclass of GMimePart, so all
> of the GMimePart API's that you are used to still work but now you also have
> access to convenient functions like g_mime_text_part_get_text() and
> g_mime_text_part_set_text() which removes the need for creating streams,
> data wrappers, and setting them on the part.
>
> 6. Brand new rfc822 address parser which is more tolerant than the previous
> generation parser. What's not to love?
>
> 7. And finally we get to a nifty feature that I just hacked up while waiting for
> some other code to compile (hey, it takes an hour to compile... I needed
> something to do!) which is that GMimeParser now scans for -----BEGIN PGP
> MESSAGE-----/-----END PGP MESSAGE----- and -----BEGIN PGP SIGNED
> MESSAGE-----/-----END PGP SIGNED MESSAGE----- markers while looking for
> MIME boundaries and sets some state on the corresponding GMimePart that
> you can use to quickly decide if the part contains encapsulated OpenPGP
> data.
>
> Jeff
>
> _______________________________________________
> balsa-list mailing list
> [hidden email]
> https://mail.gnome.org/mailman/listinfo/balsa-list
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Albrecht Dreß
In reply to this post by Gnome - Balsa mailing list
Hi Jeff:

Thank you so much for this information!

I believe your new lib will simplify many parts in Balsa (but we probably still need to support the old GMime 2.6 based stuff until all LTS distos switched to the new one).

Some comments below...

Am 13.03.17 23:38 schrieb(en) Jeffrey Stedfast via balsa-list:
> 2. Next up is the replacement of the old custom GnuPG back-end with GpgMe. Also included with this change is full support for S/MIME via both multipart/signed and application/pkcs7-mime content-types using gpgsm (via GpgMe).

That's cool!  You may recall that I never used your GMime GnuPG implementation in Balsa, but implemented my own solution on top of GpgME more than 10 years (IIRC) ago, which is working just fine afaict... ;-)

> During this change, I also took the liberty of simplifying the crypto API's a bit and so I was able to make it such that g_mime_multipart_signed_verify(), for example, no longer requires you to pass it a GMimeCryptoContext. Instead, GMime parses the Content-Type's protocol parameter and uses that to instantiate the correct crypto context (GMimeGpgContext for PGP and GMimePkcs7Context for S/MIME).

Also very nice!

> Many of the various state properties have been replaced with bitflags that can be passed to encrypt() and decrypt(). The sign() method now also takes a detach argument (might make this into a bitflag instead?) in order to support encapsulated signing.

Do you have some more documentation about this?  It would be interesting to know how this approach could fit with the current GpgME-based implementation in Balsa.

> 4. New in GMime 3.0 is the GMimeParserOptions struct which can be passed to GMimeParser and other parser functions exposed in the lower-level API's. This structure helps define how strict/forgiving the various parsing routines should be with the input. This replaces the need for g_mime_init()'s flags so you can change these settings on the fly now.

It would be *really* cool if the parser could optionally collect and provide information about violations of the various rfc's encountered during the parsing process.  Maybe something like a linked list containing the stream offset, a unique code and a comment, or something similar, with an upper limit of items collected.  This might provide valuable information for spam and/or malware checking (and would be *a lot* more performant than the perl/python based parsers typically used, I guess).  Not related to Balsa, but to an other project I'm working on...

> 6. Brand new rfc822 address parser which is more tolerant than the previous generation parser. What's not to love?

Same as above - please provide more information about the rfc violations...

> 7. And finally we get to a nifty feature that I just hacked up while waiting for some other code to compile (hey, it takes an hour to compile... I needed something to do!) which is that GMimeParser now scans for -----BEGIN PGP MESSAGE-----/-----END PGP MESSAGE----- and -----BEGIN PGP SIGNED MESSAGE-----/-----END PGP SIGNED MESSAGE----- markers while looking for MIME boundaries and sets some state on the corresponding GMimePart that you can use to quickly decide if the part contains encapsulated OpenPGP data

That's also a nice feature (and again something I implemented in Balsa more than a decade ago).  In case of a cleartext signed message, the header lines to look for are (in this order) "-----BEGIN PGP SIGNED MESSAGE-----", "-----BEGIN PGP SIGNATURE-----" and "-----END PGP SIGNATURE-----", btw. (see RFC 4880, sect. 7).  You might want to limit the check to text parts; malware spammers recently added fake OpenPGP blocks to M$ office documents containing malicious macros.

Looking forward to the new release,
Cheers,
Albrecht.
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list

attachment0 (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Upcoming GMime 3.0 changes

Gnome - Balsa mailing list
Hey Albrecht, thanks for replying!

Answers below...

> -----Original Message-----
> From: Albrecht Dreß [mailto:[hidden email]]
> Sent: Tuesday, March 14, 2017 2:14 PM
> To: Jeffrey Stedfast <[hidden email]>
> Cc: [hidden email]
> Subject: Re: Upcoming GMime 3.0 changes
>
> Hi Jeff:
>
> Thank you so much for this information!

You're welcome!

>
> I believe your new lib will simplify many parts in Balsa (but we probably still
> need to support the old GMime 2.6 based stuff until all LTS distos switched to
> the new one).
>
> Some comments below...
>
> Am 13.03.17 23:38 schrieb(en) Jeffrey Stedfast via balsa-list:
> > 2. Next up is the replacement of the old custom GnuPG back-end with
> GpgMe. Also included with this change is full support for S/MIME via both
> multipart/signed and application/pkcs7-mime content-types using gpgsm (via
> GpgMe).
>
> That's cool!  You may recall that I never used your GMime GnuPG
> implementation in Balsa, but implemented my own solution on top of
> GpgME more than 10 years (IIRC) ago, which is working just fine afaict... ;-)

Yea, I do remember this... I think it's one of the reasons that I made the GMime API's return more complete info about signers rather than just good/bad/etc for 2.6.

In the end, it just makes sense to use gpgme now due to the fact that it is now installed pretty much everywhere, even my Mac (via homebrew)!

Hopefully you'll be able to reduce your usage of gpgme directly now that GMime uses gpgme 😊

>
> > During this change, I also took the liberty of simplifying the crypto API's a bit
> and so I was able to make it such that g_mime_multipart_signed_verify(), for
> example, no longer requires you to pass it a GMimeCryptoContext. Instead,
> GMime parses the Content-Type's protocol parameter and uses that to
> instantiate the correct crypto context (GMimeGpgContext for PGP and
> GMimePkcs7Context for S/MIME).
>
> Also very nice!
>
> > Many of the various state properties have been replaced with bitflags that
> can be passed to encrypt() and decrypt(). The sign() method now also takes a
> detach argument (might make this into a bitflag instead?) in order to support
> encapsulated signing.
>
> Do you have some more documentation about this?  It would be interesting
> to know how this approach could fit with the current GpgME-based
> implementation in Balsa.

When I was moving to gpgme, I noticed that the always_trust flag was one of the gpgme_encrypt_flags_t, so it made sense to use a similar mapping (GMimeEncryptFlags). Of course, ALWAYS_TRUST is currently the only flag for encrypting while EXPORT_SESSION_KEY is the only GMimeDecryptFlags flag. Currently GMimeVerifyFlags has no enum values available (the only one that would have made sense from the old GMime API would be auto-key-retrieve but gpgme doesn't seem to have a mapping for that except, perhaps, if I use the gpgme API for manipulating the configuration, but even then I'm not sure it'd work).

>
> > 4. New in GMime 3.0 is the GMimeParserOptions struct which can be
> passed to GMimeParser and other parser functions exposed in the lower-
> level API's. This structure helps define how strict/forgiving the various
> parsing routines should be with the input. This replaces the need for
> g_mime_init()'s flags so you can change these settings on the fly now.
>
> It would be *really* cool if the parser could optionally collect and provide
> information about violations of the various rfc's encountered during the
> parsing process.  Maybe something like a linked list containing the stream
> offset, a unique code and a comment, or something similar, with an upper
> limit of items collected.  This might provide valuable information for spam
> and/or malware checking (and would be *a lot* more performant than the
> perl/python based parsers typically used, I guess).  Not related to Balsa, but
> to an other project I'm working on...

Yea, I've thought about this off and on for at least the past decade... I just haven't done it because it'd make the API's more awkward with another argument. Plus GMime is mostly focused on mail clients that need to do their best to interoperate with everything as opposed to spam traps which might want to use that info for holding mail back.

The other problem is that GMime's GMimeParser really only parses the stream into field/value header pairs and individual mime-parts. Parsing of the header values doesn't occur until later, inside of the GMimeObject implementation and so there's no way to pass that info back to the GMimeParser to give back to its caller.

I sort of did this with MimeKit (my C# re-implementation fop GMime) using exceptions. E.g. if you set the ParserOptions to use strict(er) rules, it would throw an exception giving you the offset of the beginning of the token, the offset of the error byte, and a description of the error, but C doesn't have exceptions...

>
> > 6. Brand new rfc822 address parser which is more tolerant than the
> previous generation parser. What's not to love?
>
> Same as above - please provide more information about the rfc violations...

The main thing I did was to try and work around more common brokenness, such as having addresses like:

To: Warren Worthington, Jr. <[hidden email]>

(notice the unquoted , and .)

Also worked around are things like the ones mentioned in https://tools.ietf.org/html/rfc7103#section-7

I don't think I managed to support *all* of the broken address forms listed in section 7, but I managed to add support for a number of them.

>
> > 7. And finally we get to a nifty feature that I just hacked up while waiting
> for some other code to compile (hey, it takes an hour to compile... I needed
> something to do!) which is that GMimeParser now scans for -----BEGIN PGP
> MESSAGE-----/-----END PGP MESSAGE----- and -----BEGIN PGP SIGNED
> MESSAGE-----/-----END PGP SIGNED MESSAGE----- markers while looking for
> MIME boundaries and sets some state on the corresponding GMimePart that
> you can use to quickly decide if the part contains encapsulated OpenPGP
> data
>
> That's also a nice feature (and again something I implemented in Balsa more
> than a decade ago).  In case of a cleartext signed message, the header lines
> to look for are (in this order) "-----BEGIN PGP SIGNED MESSAGE-----", "-----
> BEGIN PGP SIGNATURE-----" and "-----END PGP SIGNATURE-----", btw.

Ah! Thanks for the catch! I'll fix that up...

> (see
> RFC 4880, sect. 7).  You might want to limit the check to text parts; malware
> spammers recently added fake OpenPGP blocks to M$ office documents
> containing malicious macros.

The GMimeParser just adds those lines to the list of boundaries it checks for since it's already checking for strings starting with -- 😊

There's no harm in checking for them - it doesn't actually do anything with them.

Jeff

_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Albrecht Dreß
In reply to this post by Gnome - Balsa mailing list
Hi Jeff:

Am 14.03.17 19:47 schrieb(en) Jeffrey Stedfast via balsa-list:
> Yea, I do remember this... I think it's one of the reasons that I made the GMime API's return more complete info about signers rather than just good/bad/etc for 2.6.

Balsa uses (in libbalsa/gmime-gpgme-signature.h) a GMimeGpgmeSigstat object, collecting various gpgme data items.  IMHO it wouldn't make sense to invent new data types for things already being defined properly in gpgme.

> In the end, it just makes sense to use gpgme now due to the fact that it is now installed pretty much everywhere, even my Mac (via homebrew)!

Exactly!

> Hopefully you'll be able to reduce your usage of gpgme directly now that GMime uses gpgme 😊

The GMime interaction is wrapped into a few source files (libbalsa/gmime-*.[hc], with the exception of gmime-stream-gio.[hc]).

The latter would be a nice extension to GMime, too, in particular as the GIO streams are really powerful.  Actually, you could also implement some of the other streams on top of GIO.  Feel free to use any part of it...

Large parts of the gpgme interaction in Balsa are for proper dealing with keys and certificates (key/cert selection, data display, etc.).  As I mentioned above, IMO it would *really* simplify life if GMime would use e.g. gpgme_key_t directly.

> When I was moving to gpgme, I noticed that the always_trust flag was one of the gpgme_encrypt_flags_t, so it made sense to use a similar mapping (GMimeEncryptFlags).

I suggest to use the gpgme_encrypt_flags_t enum.  I don't see any benefit of just wrapping the existing gpgme values into a new data type!

> Of course, ALWAYS_TRUST is currently the only flag for encrypting

GPGME_ENCRYPT_NO_COMPRESS may make sense even for a MUA, at least if the message size is small (not implemented in Balsa yet, though).  Symmetric encryption should be supported if you want to keep GMime a universal tool, not strictly limited to a MUA.  Again, why re-invent the wheel? ;-)

> while EXPORT_SESSION_KEY is the only GMimeDecryptFlags flag. Currently GMimeVerifyFlags has no enum values available (the only one that would have made sense from the old GMime API would be auto-key-retrieve but gpgme doesn't seem to have a mapping for that except, perhaps, if I use the gpgme API for manipulating the configuration, but even then I'm not sure it'd work).

I fully  agree with you here, as I don't think that these options (including TOFU as the latest cool invention) should be configured on the MUA level.  But you really shouldn't limit gpgme's capabilities - if gpgme has an useful interface, you should /really/ consider to just use it, instead of doing the same just in a different way!

> Yea, I've thought about this off and on for at least the past decade... I just haven't done it because it'd make the API's more awkward with another argument. Plus GMime is mostly focused on mail clients that need to do their best to interoperate with everything as opposed to spam traps which might want to use that info for holding mail back.

Well, GMime is also a perfect toolset for dissecting malware spam...

> The other problem is that GMime's GMimeParser really only parses the stream into field/value header pairs and individual mime-parts. Parsing of the header values doesn't occur until later, inside of the GMimeObject implementation and so there's no way to pass that info back to the GMimeParser to give back to its caller.
>
> I sort of did this with MimeKit (my C# re-implementation fop GMime) using exceptions. E.g. if you set the ParserOptions to use strict(er) rules, it would throw an exception giving you the offset of the beginning of the token, the offset of the error byte, and a description of the error, but C doesn't have exceptions...

A trivial suggestion: as everything in GMime is derived from GObject, just define a signal (e.g. "parse-error") which is emitted with the appropriate data (error code, stream position if available) whenever a recoverable error occurs.  This doesn't require any additional parameters, the internal overhead is minimal, but it would add /a lot/ of value to the library.

> The main thing I did was to try and work around more common brokenness, such as having addresses like:
>
> To: Warren Worthington, Jr. <[hidden email]>
>
> (notice the unquoted , and .)
>
> Also worked around are things like the ones mentioned in https://tools.ietf.org/html/rfc7103#section-7
>
> I don't think I managed to support *all* of the broken address forms listed in section 7, but I managed to add support for a number of them.

I see.  However, I wonder if detecting (and mitigating) them is really necessary for a MUA.  Most are an indication for badly crafted spam or malware (as well as some rfc violations coming from MUA's like Lookout, which you also might want to describe as badly crafted malware ;-).  It would be highly valuable for detecting spam/malware messages, though, so the information which RFC 7103 rule has been detected would be really cool.

> Ah! Thanks for the catch! I'll fix that up...

You're welcome...

> The GMimeParser just adds those lines to the list of boundaries it checks for since it's already checking for strings starting with -- 😊
>
> There's no harm in checking for them - it doesn't actually do anything with them.

I don't think it is a security hazard passing these malware office documents into gpgme - it's extremely robust afaict.  Anyway, a MUA would probably check only for such headers in text/* parts.

Did you think about decrypting/signature checking such (text) parts in GMime?  It might be a little difficult as such parts might result in one up to three resulting parts:  optionally some unencrypted/unsigned matter before the opening header (missing if the header starts at the very beginning of the part), the signed/encrypted stuff, and optionally some extra data behind the closing header (typically if a mailing list processor adds some lines).  Balsa can only handle the case where the part starts with the opening and ends with the closing header (i.e. only one resulting part).  If GMime could deal with it, this would be cool.

Cheers,
Albrecht.
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list

attachment0 (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Upcoming GMime 3.0 changes

Gnome - Balsa mailing list
Hey Albrecht,

> -----Original Message-----
> From: Albrecht Dreß [mailto:[hidden email]]
> Sent: Tuesday, March 14, 2017 4:39 PM
> To: Jeffrey Stedfast <[hidden email]>
> Cc: [hidden email]
> Subject: Re: Upcoming GMime 3.0 changes
>
> Hi Jeff:
>
> Am 14.03.17 19:47 schrieb(en) Jeffrey Stedfast via balsa-list:
> > Yea, I do remember this... I think it's one of the reasons that I made the
> GMime API's return more complete info about signers rather than just
> good/bad/etc for 2.6.
>
> Balsa uses (in libbalsa/gmime-gpgme-signature.h) a GMimeGpgmeSigstat
> object, collecting various gpgme data items.  IMHO it wouldn't make sense to
> invent new data types for things already being defined properly in gpgme.

Except that the gpgme state gets wiped out in the very next operation you perform on the context (or if the ctx gets freed).

So it *is* useful to copy it (especially now that a # of the API's internally create their own ctx and then free it when they are done). Plus it provides an abstraction... which can be nice.

>
> > In the end, it just makes sense to use gpgme now due to the fact that it is
> now installed pretty much everywhere, even my Mac (via homebrew)!
>
> Exactly!
>
> > Hopefully you'll be able to reduce your usage of gpgme directly now that
> GMime uses gpgme 😊
>
> The GMime interaction is wrapped into a few source files (libbalsa/gmime-
> *.[hc], with the exception of gmime-stream-gio.[hc]).

I'll check them out.

>
> The latter would be a nice extension to GMime, too, in particular as the GIO
> streams are really powerful.  Actually, you could also implement some of the
> other streams on top of GIO.  Feel free to use any part of it...

GMime has actually had a gmime-stream-gio.[c,h] since at least 2.6 (also in 3.0).

>
> Large parts of the gpgme interaction in Balsa are for proper dealing with keys
> and certificates (key/cert selection, data display, etc.).  As I mentioned
> above, IMO it would *really* simplify life if GMime would use e.g.
> gpgme_key_t directly.

If there's a way to clone gpgme keys, I'll look into it... I just don't want them to get wiped out if you hold onto the keys but perform another op (or free the ctx).

Perhaps I can have a void *raw_key or something... so that you can pass that ptr directly to gpgme.

>
> > When I was moving to gpgme, I noticed that the always_trust flag was one
> of the gpgme_encrypt_flags_t, so it made sense to use a similar mapping
> (GMimeEncryptFlags).
>
> I suggest to use the gpgme_encrypt_flags_t enum.  I don't see any benefit of
> just wrapping the existing gpgme values into a new data type!

Abstraction 😊

>
> > Of course, ALWAYS_TRUST is currently the only flag for encrypting
>
> GPGME_ENCRYPT_NO_COMPRESS may make sense even for a MUA, at
> least if the message size is small (not implemented in Balsa yet, though).
> Symmetric encryption should be supported if you want to keep GMime a
> universal tool, not strictly limited to a MUA.  Again, why re-invent the wheel?
> ;-)

I'll add NO_COMPRESS.

I was actually just looking at the GPG plugin for Mail.app on Mac and apparently they have an option to enable symmetric...

Maybe it makes sense to expose for mail after all?

BTW, I can probably just map the GMimeEncryptFlags 1:1 with gpgme_encrypt_flags_t - that's what I've done with most of the other crypto enums.

>
> > while EXPORT_SESSION_KEY is the only GMimeDecryptFlags flag. Currently
> GMimeVerifyFlags has no enum values available (the only one that would
> have made sense from the old GMime API would be auto-key-retrieve but
> gpgme doesn't seem to have a mapping for that except, perhaps, if I use the
> gpgme API for manipulating the configuration, but even then I'm not sure it'd
> work).
>
> I fully  agree with you here, as I don't think that these options (including
> TOFU as the latest cool invention) should be configured on the MUA level.
> But you really shouldn't limit gpgme's capabilities - if gpgme has an useful
> interface, you should /really/ consider to just use it, instead of doing the
> same just in a different way!

I'm mostly just providing an abstraction 😊

>
> > Yea, I've thought about this off and on for at least the past decade... I just
> haven't done it because it'd make the API's more awkward with another
> argument. Plus GMime is mostly focused on mail clients that need to do their
> best to interoperate with everything as opposed to spam traps which might
> want to use that info for holding mail back.
>
> Well, GMime is also a perfect toolset for dissecting malware spam...

Hah 😊

>
> > The other problem is that GMime's GMimeParser really only parses the
> stream into field/value header pairs and individual mime-parts. Parsing of the
> header values doesn't occur until later, inside of the GMimeObject
> implementation and so there's no way to pass that info back to the
> GMimeParser to give back to its caller.
> >
> > I sort of did this with MimeKit (my C# re-implementation fop GMime) using
> exceptions. E.g. if you set the ParserOptions to use strict(er) rules, it would
> throw an exception giving you the offset of the beginning of the token, the
> offset of the error byte, and a description of the error, but C doesn't have
> exceptions...
>
> A trivial suggestion: as everything in GMime is derived from GObject, just
> define a signal (e.g. "parse-error") which is emitted with the appropriate data
> (error code, stream position if available) whenever a recoverable error
> occurs.  This doesn't require any additional parameters, the internal
> overhead is minimal, but it would add /a lot/ of value to the library.

Hmmm... perhaps that is a good way of doing it.

>
> > The main thing I did was to try and work around more common
> brokenness, such as having addresses like:
> >
> > To: Warren Worthington, Jr. <[hidden email]>
> >
> > (notice the unquoted , and .)
> >
> > Also worked around are things like the ones mentioned in
> https://tools.ietf.org/html/rfc7103#section-7
> >
> > I don't think I managed to support *all* of the broken address forms listed
> in section 7, but I managed to add support for a number of them.
>
> I see.  However, I wonder if detecting (and mitigating) them is really
> necessary for a MUA.  Most are an indication for badly crafted spam or
> malware (as well as some rfc violations coming from MUA's like Lookout,
> which you also might want to describe as badly crafted malware ;-).  It would
> be highly valuable for detecting spam/malware messages, though, so the
> information which RFC 7103 rule has been detected would be really cool.

GMime will only parse them if the GMimeRfcComplianceMode has been set to LOOSE. If you set it to STRICT, it'll still fail.

>
> > Ah! Thanks for the catch! I'll fix that up...
>
> You're welcome...

Fixed, btw, and also improved a bit so that the parser can verify that the markers come in the appropriate pairs and order.

>
> > The GMimeParser just adds those lines to the list of boundaries it checks
> for since it's already checking for strings starting with -- 😊
> >
> > There's no harm in checking for them - it doesn't actually do anything with
> them.
>
> I don't think it is a security hazard passing these malware office documents
> into gpgme - it's extremely robust afaict.  Anyway, a MUA would probably
> check only for such headers in text/* parts.
>
> Did you think about decrypting/signature checking such (text) parts in
> GMime?  It might be a little difficult as such parts might result in one up to
> three resulting parts:  optionally some unencrypted/unsigned matter before
> the opening header (missing if the header starts at the very beginning of the
> part), the signed/encrypted stuff, and optionally some extra data behind the
> closing header (typically if a mailing list processor adds some lines).  Balsa can
> only handle the case where the part starts with the opening and ends with
> the closing header (i.e. only one resulting part).  If GMime could deal with it,
> this would be cool.

I added support for the simple cases already, but not for re-assembling. After decrypting/verifying, it currently just replaces the content of the GMimePart w/o trying to merge.

I'll looking into merging but that might only be safe if the content-type is text/plain, otherwise bad stuff would happen if you merged a mailing-list footer into binary content or something.

Jeff

_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Gnome - Balsa mailing list
Looks like there’s gpgme_result_ref/unref() and gpgme_key_ref/unref()

Jeff

On 3/14/17, 5:02 PM, "balsa-list on behalf of Jeffrey Stedfast via balsa-list" <[hidden email] on behalf of [hidden email]> wrote:

    Hey Albrecht,
   
    > -----Original Message-----
    > From: Albrecht Dreß [mailto:[hidden email]]
    > Sent: Tuesday, March 14, 2017 4:39 PM
    > To: Jeffrey Stedfast <[hidden email]>
    > Cc: [hidden email]
    > Subject: Re: Upcoming GMime 3.0 changes
    >
    > Hi Jeff:
    >
    > Am 14.03.17 19:47 schrieb(en) Jeffrey Stedfast via balsa-list:
    > > Yea, I do remember this... I think it's one of the reasons that I made the
    > GMime API's return more complete info about signers rather than just
    > good/bad/etc for 2.6.
    >
    > Balsa uses (in libbalsa/gmime-gpgme-signature.h) a GMimeGpgmeSigstat
    > object, collecting various gpgme data items.  IMHO it wouldn't make sense to
    > invent new data types for things already being defined properly in gpgme.
   
    Except that the gpgme state gets wiped out in the very next operation you perform on the context (or if the ctx gets freed).
   
    So it *is* useful to copy it (especially now that a # of the API's internally create their own ctx and then free it when they are done). Plus it provides an abstraction... which can be nice.
   
    >
    > > In the end, it just makes sense to use gpgme now due to the fact that it is
    > now installed pretty much everywhere, even my Mac (via homebrew)!
    >
    > Exactly!
    >
    > > Hopefully you'll be able to reduce your usage of gpgme directly now that
    > GMime uses gpgme 😊
    >
    > The GMime interaction is wrapped into a few source files (libbalsa/gmime-
    > *.[hc], with the exception of gmime-stream-gio.[hc]).
   
    I'll check them out.
   
    >
    > The latter would be a nice extension to GMime, too, in particular as the GIO
    > streams are really powerful.  Actually, you could also implement some of the
    > other streams on top of GIO.  Feel free to use any part of it...
   
    GMime has actually had a gmime-stream-gio.[c,h] since at least 2.6 (also in 3.0).
   
    >
    > Large parts of the gpgme interaction in Balsa are for proper dealing with keys
    > and certificates (key/cert selection, data display, etc.).  As I mentioned
    > above, IMO it would *really* simplify life if GMime would use e.g.
    > gpgme_key_t directly.
   
    If there's a way to clone gpgme keys, I'll look into it... I just don't want them to get wiped out if you hold onto the keys but perform another op (or free the ctx).
   
    Perhaps I can have a void *raw_key or something... so that you can pass that ptr directly to gpgme.
   
    >
    > > When I was moving to gpgme, I noticed that the always_trust flag was one
    > of the gpgme_encrypt_flags_t, so it made sense to use a similar mapping
    > (GMimeEncryptFlags).
    >
    > I suggest to use the gpgme_encrypt_flags_t enum.  I don't see any benefit of
    > just wrapping the existing gpgme values into a new data type!
   
    Abstraction 😊
   
    >
    > > Of course, ALWAYS_TRUST is currently the only flag for encrypting
    >
    > GPGME_ENCRYPT_NO_COMPRESS may make sense even for a MUA, at
    > least if the message size is small (not implemented in Balsa yet, though).
    > Symmetric encryption should be supported if you want to keep GMime a
    > universal tool, not strictly limited to a MUA.  Again, why re-invent the wheel?
    > ;-)
   
    I'll add NO_COMPRESS.
   
    I was actually just looking at the GPG plugin for Mail.app on Mac and apparently they have an option to enable symmetric...
   
    Maybe it makes sense to expose for mail after all?
   
    BTW, I can probably just map the GMimeEncryptFlags 1:1 with gpgme_encrypt_flags_t - that's what I've done with most of the other crypto enums.
   
    >
    > > while EXPORT_SESSION_KEY is the only GMimeDecryptFlags flag. Currently
    > GMimeVerifyFlags has no enum values available (the only one that would
    > have made sense from the old GMime API would be auto-key-retrieve but
    > gpgme doesn't seem to have a mapping for that except, perhaps, if I use the
    > gpgme API for manipulating the configuration, but even then I'm not sure it'd
    > work).
    >
    > I fully  agree with you here, as I don't think that these options (including
    > TOFU as the latest cool invention) should be configured on the MUA level.
    > But you really shouldn't limit gpgme's capabilities - if gpgme has an useful
    > interface, you should /really/ consider to just use it, instead of doing the
    > same just in a different way!
   
    I'm mostly just providing an abstraction 😊
   
    >
    > > Yea, I've thought about this off and on for at least the past decade... I just
    > haven't done it because it'd make the API's more awkward with another
    > argument. Plus GMime is mostly focused on mail clients that need to do their
    > best to interoperate with everything as opposed to spam traps which might
    > want to use that info for holding mail back.
    >
    > Well, GMime is also a perfect toolset for dissecting malware spam...
   
    Hah 😊
   
    >
    > > The other problem is that GMime's GMimeParser really only parses the
    > stream into field/value header pairs and individual mime-parts. Parsing of the
    > header values doesn't occur until later, inside of the GMimeObject
    > implementation and so there's no way to pass that info back to the
    > GMimeParser to give back to its caller.
    > >
    > > I sort of did this with MimeKit (my C# re-implementation fop GMime) using
    > exceptions. E.g. if you set the ParserOptions to use strict(er) rules, it would
    > throw an exception giving you the offset of the beginning of the token, the
    > offset of the error byte, and a description of the error, but C doesn't have
    > exceptions...
    >
    > A trivial suggestion: as everything in GMime is derived from GObject, just
    > define a signal (e.g. "parse-error") which is emitted with the appropriate data
    > (error code, stream position if available) whenever a recoverable error
    > occurs.  This doesn't require any additional parameters, the internal
    > overhead is minimal, but it would add /a lot/ of value to the library.
   
    Hmmm... perhaps that is a good way of doing it.
   
    >
    > > The main thing I did was to try and work around more common
    > brokenness, such as having addresses like:
    > >
    > > To: Warren Worthington, Jr. <[hidden email]>
    > >
    > > (notice the unquoted , and .)
    > >
    > > Also worked around are things like the ones mentioned in
    > https://tools.ietf.org/html/rfc7103#section-7
    > >
    > > I don't think I managed to support *all* of the broken address forms listed
    > in section 7, but I managed to add support for a number of them.
    >
    > I see.  However, I wonder if detecting (and mitigating) them is really
    > necessary for a MUA.  Most are an indication for badly crafted spam or
    > malware (as well as some rfc violations coming from MUA's like Lookout,
    > which you also might want to describe as badly crafted malware ;-).  It would
    > be highly valuable for detecting spam/malware messages, though, so the
    > information which RFC 7103 rule has been detected would be really cool.
   
    GMime will only parse them if the GMimeRfcComplianceMode has been set to LOOSE. If you set it to STRICT, it'll still fail.
   
    >
    > > Ah! Thanks for the catch! I'll fix that up...
    >
    > You're welcome...
   
    Fixed, btw, and also improved a bit so that the parser can verify that the markers come in the appropriate pairs and order.
   
    >
    > > The GMimeParser just adds those lines to the list of boundaries it checks
    > for since it's already checking for strings starting with -- 😊
    > >
    > > There's no harm in checking for them - it doesn't actually do anything with
    > them.
    >
    > I don't think it is a security hazard passing these malware office documents
    > into gpgme - it's extremely robust afaict.  Anyway, a MUA would probably
    > check only for such headers in text/* parts.
    >
    > Did you think about decrypting/signature checking such (text) parts in
    > GMime?  It might be a little difficult as such parts might result in one up to
    > three resulting parts:  optionally some unencrypted/unsigned matter before
    > the opening header (missing if the header starts at the very beginning of the
    > part), the signed/encrypted stuff, and optionally some extra data behind the
    > closing header (typically if a mailing list processor adds some lines).  Balsa can
    > only handle the case where the part starts with the opening and ends with
    > the closing header (i.e. only one resulting part).  If GMime could deal with it,
    > this would be cool.
   
    I added support for the simple cases already, but not for re-assembling. After decrypting/verifying, it currently just replaces the content of the GMimePart w/o trying to merge.
   
    I'll looking into merging but that might only be safe if the content-type is text/plain, otherwise bad stuff would happen if you merged a mailing-list footer into binary content or something.
   
    Jeff
   
    _______________________________________________
    balsa-list mailing list
    [hidden email]
    https://mail.gnome.org/mailman/listinfo/balsa-list
   

_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Gnome - Balsa mailing list
In reply to this post by Albrecht Dreß
So I took a look at the gpgme code in libbalsa, specifically the code that deals with prompting the user about selecting a key to use for signing and/or whether or not to trust a low-validity key.

It looks like this could largely be done *before* calling GMime methods.

In the case of selecting a key, you could simply pass the fingerprint (instead of the email address) if the key as the userid argument to g_mime_*sign().

The downside is that it requires some code duplication wrt key lookup, but that’s fairly minimal…

I could possibly add a GMimeSelectKeyCallback to GMimeCryptoContext or something for the select key thing…

I could possibly do something similar for making sure that the user wants to encrypt to a low-validity key.

Thoughts?

Jeff


_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Albrecht Dreß
In reply to this post by Gnome - Balsa mailing list
Hi Jeff:

Am 14.03.17 22:02 schrieb(en) Jeffrey Stedfast via balsa-list:
> GMime has actually had a gmime-stream-gio.[c,h] since at least 2.6 (also in 3.0).

Ooops.  That one missed my attention, as it apparently never made it into the official docs.  Neither on the official web page <https://developer.gnome.org/gmime/stable/Streams.html>, nor in the dpcs package on my Ubuntu 16.04 LTS system (libgmime-2.6-doc ver. 2.6.20-1; <http://packages.ubuntu.com/xenial/all/libgmime-2.6-doc/filelist>).  Might this be a mistake, or an error in the Makefile?

Anyway, it's good news, as we can remove the redundant implementation from Balsa!

> If there's a way to clone gpgme keys, I'll look into it... I just don't want them to get wiped out if you hold onto the keys but perform another op (or free the ctx).

You can ref and unref them (see <https://www.gnupg.org/documentation/manuals/gpgme/Manipulating-Keys.html#index-gpgme_005fkey_005fref>).

> I was actually just looking at the GPG plugin for Mail.app on Mac and apparently they have an option to enable symmetric...
>
> Maybe it makes sense to expose for mail after all?

Hmm, might also make sense for Balsa. I'll have to think about the user interface...

>> A trivial suggestion: as everything in GMime is derived from GObject, just define a signal (e.g. "parse-error") which is emitted with the appropriate data (error code, stream position if available) whenever a recoverable error occurs.  This doesn't require any additional parameters, the internal overhead is minimal, but it would add /a lot/ of value to the library.
>
> Hmmm... perhaps that is a good way of doing it.

I can only stress thit it would be really helpful!

> GMime will only parse them if the GMimeRfcComplianceMode has been set to LOOSE. If you set it to STRICT, it'll still fail.

Actually, I wonder if there is a real use case for the strict mode.

I think a MUA will typically try to extract as much usable information as possible.  And probably display a warning to the user that the contents was somewhat broken, so the result might differ from what the sender intended to show.  This can be achieved easily by catching the parser error signals.

And my analysis tool of course /also/ wants to extract as much information as possible, so further checks (like semantic analysis) can be done.  The count and types of error signals occurring would be valuable input for the rating mechanism.

> I added support for the simple cases already, but not for re-assembling. After decrypting/verifying, it currently just replaces the content of the GMimePart w/o trying to merge.

I'm not sure if I fully understand your approach here.  Assuming you get the following text/plain part:

<snip>
Some text.
-----BEGIN PGP SIGNED MESSAGE-----
More text.
-----BEGIN PGP SIGNATURE-----
[Correct Signature]
-----END PGP SIGNATURE-----
Some Footer.
</snip>

would your code replace it by a /single/ text part, containing

<snip>
Some text.
More text.
Some Footer.
</snip>

plus the meta-information that it had a valid RFC4880 signature?

This would be *wrong* as not the complete part is protected!  Just dropping the stuff not covered by the signature would also be wrong as information is lost.  IMO the only solution is to create a chain of three text/plain chunks in this case, containing the first, second and third line, respectively, and only the 2nd part would have an associated signature info.

You may think of even more weird cases, like multiple signed/encrypted blocks within a single text part.  However, a not too rare situation is a footer being appended outside the signed matter, i.e. only the 2nd and the 3rd part would exist in the example above.

> I'll looking into merging but that might only be safe if the content-type is text/plain, otherwise bad stuff would happen if you merged a mailing-list footer into binary content or something.

IMO its safe to limit this approach to text parts.

Cheers,
Albrecht.
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list

attachment0 (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Albrecht Dreß
In reply to this post by Gnome - Balsa mailing list
Hi Jeff:

Am 15.03.17 01:55 schrieb(en) Jeffrey Stedfast via balsa-list:
> So I took a look at the gpgme code in libbalsa, specifically the code that deals with prompting the user about selecting a key to use for signing and/or whether or not to trust a low-validity key.
>
> It looks like this could largely be done *before* calling GMime methods.

Yes.  Actually it /should/ be done before!  It also offers new options, like linking preferred key id's to users in the address book (as in Kontact/Kmail).

> In the case of selecting a key, you could simply pass the fingerprint (instead of the email address) if the key as the userid argument to g_mime_*sign().

Or, as the bulk of work (GUI interaction etc.) has to be done by the caller anyway, as a GList of gpgme_key_t items -   which saves you the (costly) lookup operation!

> I could possibly add a GMimeSelectKeyCallback to GMimeCryptoContext or something for the select key thing…

IMO, there are three different situations when we pass a list of user-id's (i.e. mail addresses) to GMime:
- There is exactly /one/ matching key for the user reported by gpgme.  No callback is necessary.
- Gpgme reports multiple matching keys.  Even if the callback returns the list of matching keys, the caller still has to deal with the GUI, including the option to cancel the whole process.
- There are no matching keys, e.g. because the message shall be sent to an alias which is not included in the key's user id's.  In this case, the caller must list /all/ keys, and again the process could be cancelled.

I think if the GMime encrypt function would expect just a list of gpgme_key_t's, which shall be assembled by the caller, this would simplify everything a lot, inter alia eliminating the cancellation path from the callback.

Listing keys/certificates is an expensive operation, btw, in particular if the key ring is large, or if the cert store is slow (like a remote ldap server).  The callback approach might require the same operation several times, which could actually add a considerable delay.

So, you are right, all the key selection work should be done /before/ calling GMime!

> I could possibly do something similar for making sure that the user wants to encrypt to a low-validity key.

Again, I think this should be done before calling GMime.  If the caller includes untrusted keys in the gpgme_key_t GList and doesn't set the proper option for the GMime method, just fail with an appropriate error code.

A callback might be called multiple times if the GList contains more than one untrusted key which is tedious.  If the caller is responsible, the GUI could be optimised (e.g. in a dialogue like "The following n keys are untrusted: <show the complete list>").

Cheers,
Albrecht.
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list

attachment0 (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Gnome - Balsa mailing list
In reply to this post by Albrecht Dreß
Sadly Outlook sucks at quoting replies… so forgive me for the suckage (

On 3/15/17, 2:14 PM, "Albrecht Dreß" <[hidden email]> wrote:

    Hi Jeff:
   
    Am 14.03.17 22:02 schrieb(en) Jeffrey Stedfast via balsa-list:
    > GMime has actually had a gmime-stream-gio.[c,h] since at least 2.6 (also in 3.0).
   
    Ooops.  That one missed my attention, as it apparently never made it into the official docs.  Neither on the official web page <https://developer.gnome.org/gmime/stable/Streams.html>, nor in the dpcs package on my Ubuntu 16.04 LTS system (libgmime-2.6-doc ver. 2.6.20-1; <http://packages.ubuntu.com/xenial/all/libgmime-2.6-doc/filelist>).  Might this be a mistake, or an error in the Makefile?


Looks like I never added it to the docs :-\

I’ll have to fix that…

   
    Anyway, it's good news, as we can remove the redundant implementation from Balsa!
   
    > If there's a way to clone gpgme keys, I'll look into it... I just don't want them to get wiped out if you hold onto the keys but perform another op (or free the ctx).
   
    You can ref and unref them (see <https://www.gnupg.org/documentation/manuals/gpgme/Manipulating-Keys.html#index-gpgme_005fkey_005fref>).


Yea, I knew about key refs, but not the result refs.

   
    > I was actually just looking at the GPG plugin for Mail.app on Mac and apparently they have an option to enable symmetric...
    >
    > Maybe it makes sense to expose for mail after all?
   
    Hmm, might also make sense for Balsa. I'll have to think about the user interface...
   
    >> A trivial suggestion: as everything in GMime is derived from GObject, just define a signal (e.g. "parse-error") which is emitted with the appropriate data (error code, stream position if available) whenever a recoverable error occurs.  This doesn't require any additional parameters, the internal overhead is minimal, but it would add /a lot/ of value to the library.
    >
    > Hmmm... perhaps that is a good way of doing it.
   
    I can only stress thit it would be really helpful!
   
    > GMime will only parse them if the GMimeRfcComplianceMode has been set to LOOSE. If you set it to STRICT, it'll still fail.
   
    Actually, I wonder if there is a real use case for the strict mode.


I mostly have strict mode so I can satisfy my OCD about having a strict parser (

The defaults are all LOOSE.

   
    I think a MUA will typically try to extract as much usable information as possible.  And probably display a warning to the user that the contents was somewhat broken, so the result might differ from what the sender intended to show.  This can be achieved easily by catching the parser error signals.
   
    And my analysis tool of course /also/ wants to extract as much information as possible, so further checks (like semantic analysis) can be done.  The count and types of error signals occurring would be valuable input for the rating mechanism.
   
    > I added support for the simple cases already, but not for re-assembling. After decrypting/verifying, it currently just replaces the content of the GMimePart w/o trying to merge.
   
    I'm not sure if I fully understand your approach here.  Assuming you get the following text/plain part:
   
    <snip>
    Some text.
    -----BEGIN PGP SIGNED MESSAGE-----
    More text.
    -----BEGIN PGP SIGNATURE-----
    [Correct Signature]
    -----END PGP SIGNATURE-----
    Some Footer.
    </snip>
   
    would your code replace it by a /single/ text part, containing
   
    <snip>
    Some text.
    More text.
    Some Footer.
    </snip>
   
    plus the meta-information that it had a valid RFC4880 signature?


No, what I mean is that my current code would replace the existing GMimePart’s content with just what is armored and the rest would be dropped. In other words, “Some text.\n” and “Some footer.” Would not exist once you verified the part.

The code does not add any meta-information to the GMimePart content (that would be very client-specific in the way it decided to display that info).

Instead, the verify() method simply returns a list of signatures.

I thought what you wanted from your earlier description was the <snip> section you pasted above, but now it sounds like what you actually want is for g_mime_part_openpgp_verify() is to return a GMimeMultipart of 3 text/plain parts (the lead, the verified text, and the footer).

Is that correct?

That sounds awkward, especially once you start handling multiple armored sections in the original text/plain part.

I guess this kills the usefulness of my current implementation as well, though, since the header/footer text would get lost.

I guess I had assumed that there wasn’t likely to be a header and any footer that existed would just be mailing-list munging which I doubt anyone cares much about keeping.

Oh well…

Jeff

_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Albrecht Dreß
Hi Jeff:

Am 15.03.17 19:58 schrieb(en) Jeffrey Stedfast via balsa-list:
> Sadly Outlook sucks at quoting replies… so forgive me for the suckage (

No problem, I can identify your answers...

> Looks like I never added it to the docs :-\
>
> I’ll have to fix that…

For Balsa, it's done with changing exactly three lines, plus removing two files...

> I mostly have strict mode so I can satisfy my OCD about having a strict parser (
>
> The defaults are all LOOSE.

I see.

> No, what I mean is that my current code would replace the existing GMimePart’s content with just what is armored and the rest would be dropped. In other words, “Some text.\n” and “Some footer.” Would not exist once you verified the part.
>
> The code does not add any meta-information to the GMimePart content (that would be very client-specific in the way it decided to display that info).
>
> Instead, the verify() method simply returns a list of signatures.

Sorry, I was unclear here - of course I referred to the verification status here, and mixed up the part representation in GMime vs. Balsa.

> I thought what you wanted from your earlier description was the <snip> section you pasted above, but now it sounds like what you actually want is for g_mime_part_openpgp_verify() is to return a GMimeMultipart of 3 text/plain parts (the lead, the verified text, and the footer).
>
> Is that correct?

I was thinking of a way to get the verification result of the armoured part *and* an information if it covers the whole part.  This is the behaviour of KMail, which keeps all text in this case, but marks the part of the text/plain which has actually been signed.  Please excuse me if my description was unclear...

Thinking again, three text/plain parts is also not the proper approach, as from the MIME pov it's still /one/ part.

Maybe it would be possible to glue the lead, the verified part, and the footer together in the result text buffer, and to add extra fields in the verification status indicating the start and end offset of the armoured section in it (in the typical case, the start would be 0, and the end the text length).

This would be absolutely sufficient for the KMail-like display.

> That sounds awkward,

Yes, it is.  In particular as RFC 4880 has never been designed for such a use case.  With the proper method (i.e RFC 3156) these problems can never occur!

> especially once you start handling multiple armored sections in the original text/plain part.

These are really pathological cases.  As a pragmatic approach, you could treat just the first section as above, and keep the remainder "as is".  It's then up to the caller to verify or decrypt again, until no armoured section is left.

I never saw such a message yet, btw., and I don't know how other MUA's would deal with it...

> I guess this kills the usefulness of my current implementation as well, though, since the header/footer text would get lost.
>
> I guess I had assumed that there wasn’t likely to be a header and any footer that existed would just be mailing-list munging which I doubt anyone cares much about keeping.

IMO you could keep things simple here, too.  A RFC 4880 block with extra text is actually a misuse of the standard, and a mailing list processor adding text is somewhat broken.  Just add a documentation about the limitations, and don't waste too much brains for a really weird case.

Cheers,
Albrecht.
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list

attachment0 (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Peter Bloomfield
In reply to this post by Albrecht Dreß
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi All:

On 03/14/2017 02:14:18 PM Tue, Albrecht Dreß wrote:
> Hi Jeff:
>
> Thank you so much for this information!
>
> I believe your new lib will simplify many parts in Balsa (but we probably still need to support the old GMime 2.6 based stuff until all LTS distos switched to the new one).

I installed GMime 3.0.1 from git, and I have a patch against Balsa git master for building with it, keeping the ability to build with 2.6 through the magic of #if. It just adapts Balsa's current code to work with the new API, but even so it's a big mess, over 2k lines of patch, touching 32 files and creating 160 blocks of conditionally compiled code. But it compiles and runs in at least one configuration, so it's available for exploring the new features that Jeff put in 3.0.

If we're going to need a single tree that can build with either 2.6 or 3.0, I could commit this patch. It looks first for 3.0, with a fallback to 2.6--no configure option for choosing, although one could be implemented. Alternatively, we could make a stable gmime-2.6 branch and make master 3.0-only, which would make for a more convenient development process.

Since no GMime 3.0 packages have yet been released(?), and development is possible only for the intrepid git user, another possibility would be to open a bug and post patches there, until released packages enable easier testing.

Any suggestions about how to proceed?

Peter
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAllK0QQACgkQH1/UtbkqdPWangCfWRoeXDTIP1jS39YwI1LKebYF
yOYAn3yTAaxv/NKGncKqkDjVmeaqOXzu
=vi+P
-----END PGP SIGNATURE-----
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Albrecht Dreß
Hi Peter:

Am 21.06.17 22:03 schrieb(en) Peter Bloomfield:
> I installed GMime 3.0.1 from git, and I have a patch against Balsa git master for building with it, keeping the ability to build with 2.6 through the magic of #if. It just adapts Balsa's current code to work with the new API, but even so it's a big mess, over 2k lines of patch, touching 32 files and creating 160 blocks of conditionally compiled code.

Ouch.  It's very interesting and encouraging to hear that you succeeded in compiling the sources.  But I strongly advise against using it through preprocessor conditionals.  It basically makes the code unreadable and unmaintainable, and has always been a source for funny (more or less, actually) errors.  Better create a new branch...

> But it compiles and runs in at least one configuration, so it's available for exploring the new features that Jeff put in 3.0.

Which new features are you referring to?  I.e. what was missing in 2.6 for Balsa which is new in 3.0?

> If we're going to need a single tree that can build with either 2.6 or 3.0, I could commit this patch.

See above.  *Please* don't do that!

> It looks first for 3.0, with a fallback to 2.6--no configure option for choosing, although one could be implemented. Alternatively, we could make a stable gmime-2.6 branch and make master 3.0-only, which would make for a more convenient development process.

Why would we want to have an option for 3.0 in the wild?  Is there *any* disto which ships 3.0?  Brand-new Debian stretch (which typically will be the next Ubuntu LTS) comes with 2.6.22...

And, apart from that, what would be the /technical/ reason for porting now?  If I look at Jeff's announcement regarding the new features (I omit the pure API changes):

- Internal raw value header cache.  Really useful for other purposes, but is this of any importance for Balsa?  I don't think so.
- Crypto uses GpgME.  We have it in Balsa since more than a decade, completely bypassing the old gmime gpg interface, and it works perfectly.  We could remove or shrink some files, but does this justify the effort at this point?
- Tolerant address parser.  Again useful for other purposes.  But has this /ever/ been an issue in Balsa?
- GMime detects RFC 4880 in text parts.  Balsa does it since ages (using RFC 4880 is discouraged, though).

Did I miss something?

So, IMHO there would be no benefit at the expense of less readable, less maintainable and more error prone code.  This is definitely something which should *not* go into the master branch.

> Since no GMime 3.0 packages have yet been released(?), and development is possible only for the intrepid git user, another possibility would be to open a bug and post patches there, until released packages enable easier testing.
>
> Any suggestions about how to proceed?

I think your test code is a really very interesting feasibility study.  It might be an idea to create an "experimental" branch with it, so it's not forgotten.  But I think it is *not* suitable for any community package at this point.

If we want to make Balsa more popular, IMO we have completely different problems.  I don't watch other distos, but Debian and Ubuntu all ship with some 2.4.12 variant.  But this one will just stop sending messages at some (near?) point in the future as it relies on the ancient libesmtp library which supports (insecure) SSL 3 and TLS 1.0 only, so MTA's will probably simply refuse to talk to Balsa.

Package maintainers *hopefully* now look at Balsa's master branch, iirc you sent an announcement last year.  Again changing master to something relying on non-standard packages is an extremely bad idea and will encourage maintainers not to put too much effort in something which looks like a quite unstable and experimental project.  Until Balsa disto packages silently die for the reasons above.

Looking at my personal agenda, I think the following points should be addressed shortly:
- Some improvements in the crypto code, in particular replacing the clumsy gpg code for talking to key servers by GpgME.  Goal: remove gpg config dependency, better user experience, preparation for EasyGPG which will rely on Gpgme keyserver interaction (see <https://wiki.gnupg.org/EasyGpg2016>).  I'm working on that.
- Replace the low-level networking parts in libimap, depending on openssl, by libnetclient.  Probably much work, I'll look into it after finishing the first task.  Goal: drop openssl dependency (and its frequent security issues), as everything will then be in gio (i.e GnuTLS).

And, if everything has been tested, it would be time for a new stable release IMO, which should be promoted for inclusion into distos.

Oh, and I think we should also look into the code quality (see e.g. <https://www.securecoding.cert.org/confluence/display/c/SEI+CERT+C+Coding+Standard> and <http://www.embedded.com/electronics-blogs/beginner-s-corner/4023981/Introduction-to-MISRA-C>), documentation and coding style.  Static analysis (FlexeLint or similar) would be great.  Not visible to the user, but improves stability and security.

Just my € 0.01, though, as always...

Cheers,
Albrecht.
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list

attachment0 (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Upcoming GMime 3.0 changes

Peter Bloomfield
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Albrecht,

On 06/22/2017 03:26:16 PM Thu, Albrecht Dreß wrote:
> Hi Peter:
>
> Am 21.06.17 22:03 schrieb(en) Peter Bloomfield:
>> I installed GMime 3.0.1 from git, and I have a patch against Balsa git master for building with it, keeping the ability to build with 2.6 through the magic of #if. It just adapts Balsa's current code to work with the new API, but even so it's a big mess, over 2k lines of patch, touching 32 files and creating 160 blocks of conditionally compiled code.
>
> Ouch.  It's very interesting and encouraging to hear that you succeeded in compiling the sources.  But I strongly advise against using it through preprocessor conditionals.  It basically makes the code unreadable and unmaintainable, and has always been a source for funny (more or less, actually) errors.  Better create a new branch...

Agreed.

> If we want to make Balsa more popular, IMO we have completely different problems.  I don't watch other distos, but Debian and Ubuntu all ship with some 2.4.12 variant.  But this one will just stop sending messages at some (near?) point in the future as it relies on the ancient libesmtp library which supports (insecure) SSL 3 and TLS 1.0 only, so MTA's will probably simply refuse to talk to Balsa.
>
> Package maintainers *hopefully* now look at Balsa's master branch, iirc you sent an announcement last year.  Again changing master to something relying on non-standard packages is an extremely bad idea and will encourage maintainers not to put too much effort in something which looks like a quite unstable and experimental project.  Until Balsa disto packages silently die for the reasons above.
>
> Looking at my personal agenda, I think the following points should be addressed shortly:
> - Some improvements in the crypto code, in particular replacing the clumsy gpg code for talking to key servers by GpgME.  Goal: remove gpg config dependency, better user experience, preparation for EasyGPG which will rely on Gpgme keyserver interaction (see <https://wiki.gnupg.org/EasyGpg2016>).  I'm working on that.
> - Replace the low-level networking parts in libimap, depending on openssl, by libnetclient.  Probably much work, I'll look into it after finishing the first task.  Goal: drop openssl dependency (and its frequent security issues), as everything will then be in gio (i.e GnuTLS).
>
> And, if everything has been tested, it would be time for a new stable release IMO, which should be promoted for inclusion into distos.

Looks like a plan!

> Oh, and I think we should also look into the code quality (see e.g. <https://www.securecoding.cert.org/confluence/display/c/SEI+CERT+C+Coding+Standard> and <http://www.embedded.com/electronics-blogs/beginner-s-corner/4023981/Introduction-to-MISRA-C>), documentation and coding style.  Static analysis (FlexeLint or similar) would be great.  Not visible to the user, but improves stability and security.

For the cosmetic part, we could uncrustify the whole tree. Improving code quality will take longer!

Best,

Peter
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAllNJQ8ACgkQH1/UtbkqdPUIbQCfUWdDACdfbm4uA98InnUjP9rG
f9YAn3pYSK/CEALuEjZeZ1iUsDB0crXi
=wg7Z
-----END PGP SIGNATURE-----
_______________________________________________
balsa-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/balsa-list
Loading...