Bindings improvements

[h-vetinari]

A lot of updates in #104, but some work outstanding:

• figure out detection for libimagequant on windows
  • also needs to figure out issue in v9.0.0 broke pytorch #108 regarding llvm-openmp vs. intel-openmp; dependency was disabled in pillow v9.0.1 #107
  • also, there might be some issues around tigher-than-pillow GPL license, might need a separate output...?
• figure out detection of WEBPMUX on windows
• figure out detection of XCB on windows
• use conda-forge packaged raqm, blocked on WIP: add libraqm staged-recipes#17457

[h-vetinari]

@conda-forge/pillow @conda-forge/core

#104 added some more support for the bindings that pillow offers, among them libimagequant. Apparently pillow itself does not distribute that library due to a tighter license:

Libimagequant is licensed GPLv3, which is more restrictive than the Pillow license, therefore we will not be distributing binaries with libimagequant support enabled.

... which lead to python-pillow/Pillow#6019 being raised upstream.

I see at least three options and wanted to ask for opinions:

1. simply drop libimagequant support
2. keep libimagequant as dependency and accept tighter license(-in-aggregate)
3. turn into multi-output recipe
   • single output per binding: likely too painful considering the number of options
   • pillow (default) & pillow-no-<x> (where demand exists)
   • probably needs a meta-package (otherwise pillow-no-<x> can never replace pillow as a transitive dependency). This is further complicated by already occupying (most likely) a part of the build string with Package pillow-simd? #103.

CC @radarhere

[nulano]

I did not see the Raqm discussion in #104 until now, and can't tell if it has been resolved. For reference, Pillow wheels are built with this pip command:

    pip wheel $(pip_opts) \
        --global-option build_ext --global-option --enable-raqm \
        --global-option --vendor-raqm --global-option --vendor-fribidi \
        -w $abs_wheelhouse --no-deps .

https://github.com/python-pillow/pillow-wheels/blob/a57b83c1b536d53d7c03d97200e81a9fd4bd123a/config.sh#L144-L147

[nulano]

use conda-forge packaged raqm, blocked on conda-forge/staged-recipes#17457

I don't know if this is a good idea. The reason the vendored version of Raqm is used is due to FriBiDi's license, similar to the libimagequant issue. The patch allows Raqm to be enabled optionally when users don't mind the LGPL license requirements (by loading FriBiDi at runtime if it is found). Ideally it would be an optional dependency; I'm not sufficiently familiar with Conda to know what is the best way to achieve that.

[ocefpaf]

Sadly conda doesn't have an implementation for optional dependencies. @h-vetinari at this point you are the pillow conda pkg expert and I'll defer to what you think is the best course of action here.

[h-vetinari]

I haven't reviewed all the licenses, but #104 did include fribidi as well.

It seems to me that a 4th solution might be:
4. pillow_<ver>_no_gpl (only core bindings, license-compatible), pillow_<ver>_all (also includes (L)GPL bindings)

That might also be easiest to implement. I'd appreciate if someone could tell me which bindings are compatible with the (non-standard) pillow license - for example, https://pillow.readthedocs.io/en/stable/installation.html#external-libraries doesn't mention the fribidi license implication of raqm.

I don't know if this is a good idea. The reason the vendored version of Raqm is used is due to FriBiDi's license, similar to the libimagequant issue.

I don't understand this - how is it different (functionally as well as from a license perspective) if raqm is vendored or comes in as a dependency? I had tried the installation invocation with -vendor-raqm in #104 without success, hence starting to package it directly.

[nulano]

AFAIK all dependencies have a license similar to Pillow (i.e. non-copyleft, most are BSD-style) except for libimagequant (GPL) and fribidi (LGPL). All dependencies except for these two are also included in wheels distributed by Pillow.

I don't understand this - how is it different (functionally as well as from a license perspective) if raqm is vendored or comes in as a dependency?

Raqm itself is MIT licensed. The patch in the vendored version is only to allow loading fribidi at runtime (without creating a hard dependency for the system loader/linker). This way Pillow doesn't actually distribute LGPL licensed fribidi, meaning it is not "infected" by its LGPL license. Quoting from the (L)GPL FAQ:

Does the LGPL have different requirements for statically vs dynamically linked modules with a covered work? (#LGPLStaticVsDynamic)
For the purpose of complying with the LGPL (any extant version: v2, v2.1 or v3):

(1) If you statically link against an LGPLed library, you must also provide your application in an object (not necessarily source) format, so that a user has the opportunity to modify the library and relink the application.

(2) If you dynamically link against an LGPLed library already present on the user's computer, you need not convey the library's source. On the other hand, if you yourself convey the executable LGPLed library along with your application, whether linked with statically or dynamically, you must also convey the library's sources, in one of the ways for which the LGPL provides.

IIUC (IANAL) including the LGPLed fribidi library in Pillow wheels would count as static linking (even if it is done via an SO object). Either way, if Pillow wheel distributed fribidi, Pillow and anyone who further redistributes the wheels would have to abide by LGPL restrictions (by providing fribidi sources). By being loaded at runtime, the LGPL license no longer applies, as it is clearly being loaded from the user's computer. If someone develops an application that depends on Pillow with Raqm, they are also free to redistribute them both including fribidi, if they choose to abide by LGPL.

Now, IIUC conda deals with dependencies using dynamic linking by supplying each shared library object separatly. To be honest, I have no idea how this kind of dependency interacts with the LGPL, so I'm afraid I can't help with that, especially as I'm not sufficiently familiar with how conda is typically used.

However, I expect the Raqm patch should be able to pick up fribidi if it is installed among the other shared library objects even if it is not explicitly listed in the dependencies, which I would expect is good enough to avoid falling under the terms of LGPL.

[h-vetinari]

Yeah OK, I see how including it in the wheel or not makes a difference for pillow, but conda does not understand or accept the concept of a user-supplied library. So anyone wanting to use raqm through conda-forge will require fribidi from conda-forge, and therefore incur that license being present within their environment.

AFAIK all dependencies have a license similar to Pillow (i.e. non-copyleft, most are BSD-style) except for libimagequant (GPL) and fribidi (LGPL)

Thanks. That means a possible split could be:

• pillow_<vers>_no_gpl:
  • freetype (under custom license)
  • jpeg (another custom license)
  • openjpeg (BSD-2-Clause)
  • lcms2 (MIT)
  • libtiff (customized HPND)
  • libwebp (BSD-3-Clause)
  • libxcb (MIT)
  • tk (custom license, apparently BSD)
  • zlib (another custom license)
• pillow_<vers>_all:
  • all the above, plus
  • fribidi (LGPL-2.1)
  • libimagequant (GPL-3.0-or-later)
  • raqm (MIT, but depends on fribidi)

[h-vetinari]

Could someone from @conda-forge/core chime in on the proposed split into pillow_<version>_no_gpl and pillow_<version>_all (based on the license of the libraries for which we include bindings), and making that split by means of a build-string?

I could implement that, but haven't heard back if this is considered a good idea in principle (alternative would be dropping a bunch of bindings or accepting the GPL-in-aggregate, both of which I consider inferior), or how it should be done technically (I think build string is the best/easiest, but there are others).

[hmaarrfk]

I think we should consolidate with: conda-forge/python-feedstock#387

[h-vetinari]

Fair point, but then, that issue hasn't seen any activity in 1.5yrs, and the issue for pillow seems more urgent to several people. If there's ever something like a conda-wide metapackage to avoid the GPL, this could be easily incorporated with my proposal, but I think it's not necessary to couple them and IMO it would be unreasonable to require a conda-forge-wide solution just for this particular issue.

[hmaarrfk]

Honestly, I would just remove libimagequant and tell Pillow that without a better build system, they shouldn't add depeendecies like that.

They could make it a plugin that is used to compile a .so that is brought it upon request.

[hmaarrfk]

The truth is that dependencies have consequences, one consequence might be that a package isn't released as widely as it would otherwise be.

Other consequences could be that downstream patches out your system and you get inconsistent performance.

[hmaarrfk]

This would mostly get you time to discuss with conda-forge wide about how to go about gpl/non-gpl.