Better grouping of imports

[quarl]

pyflyby makes imports nice when adding to an existing import line. E.g. if "from m1 import x" exists, and "from m1 import y" needs to be added, then it merges it into "from m1 import x, y".
Pyflyby also sorts import line within one "block".
Currently it considers a blank line to separate import blocks and doesn't merge such blocks.

However, code that's carelessly maintained tends to drift into states like this:

import numpy

from pkg1.mod1 import foo
from pkg1.mod2 import bar
from pkg2 import baz
import yy

from pkg1.mod1 import foo2
from pkg1.mod3 import quux
from pkg2 import baar
import sympy
import zz

We should change the algorithm to automatically beautify this. Pyflyby has already trained people to ignore whatever's in the import stanza. So pyflyby should take greater ownership of it.

Proposal1.

• input blank lines don't separate blocks (but comments still do)
• when outputting import lines, group them like this:
  • consider imports from the same package to be one group (e.g. "import pkg1.blah", "from pkg1 import x", "from pkg1.mod1 import x")
  • output all the imports in lexicographic order, with a blank line between groups that have more than 1 import statement. So pkg1 gets special treatment but pkg2 doesn't.

We would get:

import numpy
import sympy

from pkg1.mod1 import foo, foo2
from pkg1.mod2 import bar
from pkg1.mod3 import quux

from pkg2 import baz, baar
import yy
import zz

Proposal2.
Some people like to have pkg1 show up after all the other imports, because they work in pkg1, or have stdlib imports show up before all the other imports.

Add support for something like this:

In ~/.pyflyby/*.py:
__import_sections__ = [ ['os','sys'], '*', 'pkg1']

These proposals are not mutual exclusive.

Proposal1 is higher priority than Proposal2.

[asmeurer]

https://pycqa.github.io/isort/

[asmeurer]

isort has a lot of customization (see https://pycqa.github.io/isort/docs/configuration/custom_sections_and_ordering.html and https://github.com/PyCQA/isort/blob/main/isort/settings.py#L137). The more customization we would want to adopt, the more work this will be. One must also consider that we'd have to convert the isort functions to use the pyflyby internal classes representing imports and other Python code constructs. [time estimate: 50 hrs]

[quarl]

Sounds good to proceed (sorry I missed that this was waiting for feedback)

[Carreau]

A couple of notes following #287 as there was some issues with isort.:

• input blank lines don't separate blocks (but comments still do)

I'm not sure this will work in practice, as from the top example shows:

input

import numpy

from pkg1.mod1 import foo
from pkg1.mod2 import bar
from pkg2 import baz
import yy

from pkg1.mod1 import foo2
from pkg1.mod3 import quux
from pkg2 import baar
import sympy
import zz

Output:

import numpy
import sympy

from pkg1.mod1 import foo, foo2
from pkg1.mod2 import bar
from pkg1.mod3 import quux

from pkg2 import baz, baar
import yy
import zz

The developer means to have (numpy, scipy) in it's own group.

If we do follow the logic proposed, then the output lexicographically ordered will be the following with "sympy" relagated into simple imports at the bottom.

import numpy

from pkg1.mod1 import foo, foo2
from pkg1.mod2 import bar
from pkg1.mod3 import quux

from pkg2 import baz, baar
import sympy
import yy
import zz

There are also questions with respect to comments, do the comments separate two grouping import, or do they mark a single block. That is to say:

import A
import D
import G

# comment
import B
import F
import J

import C
import E
import Z

Should we

• [ ] merge ADG, and CEZ (see below this is impossible)
• merge BFJ and CEZ

I'm inclined to go with the second and assume the second group of blank lines is a group separation in most case.

[Carreau]

So from reflexion and trying we can't merge ADG, and CEZ, reason being we need the sorting process and insertion of blank lines to be stable.

So a blocks like

# comment
import B
import B.A
import B.B
import F.A
import F.B
import J

Will after reformat have blanks lines due to the ..., with a blank line between groups that have more than 1 import statement.... requirements, becoming

# comment
import B
import B.A
import B.B

import F.A
import F.B

import J

Thus we need the comment to apply to multiple blocks with blank lines.

[Carreau]

I'm guessing there might be a need to special case from __future__ imports, those are usually always followed by a new line shouldn't they ?

[sac111gp]

I'm guessing there might be a need to special case from future imports, those are usually always followed by a new line shouldn't they ?

yes.

Any further update on this PR?

[Carreau]

#329 added the --experimental-sort-imports and should have been released.

I don't remember if it it unconditionally adds blank lines after future but will check.