FR: Support git status-based working copy snapshotting

[arxanas]

Currently, jj status crawls the filesystem itself or uses an fsmonitor (currently only Watchman). We could add another fsmonitor that calls git status to get status information. Advantages:

• Would handle submodules (if unchanged).
• Would handle Git LFS (if unchanged).
• Would let us work around bugs like Spurious jj status/jj diff due to case-insensitivity, such as for gecko-dev #1737
• Would let us use the user's custom Git fsmonitor hook, if they have one.

Additional context

Here's some issues that this might help (or have helped) to resolve/work around (to various degrees):

• Speed up .gitignores #8
• Add support for .gitattributes #53
• Global .gitignore not respected #87
• Add support for Git LFS #80
• Global .gitignore not respected #87
• Slow jj status with large numbers of ignored files #247
• Auto-add of untracked files screws me up every time #323
• Spurious jj status/jj diff due to case-insensitivity, such as for gecko-dev #1737
• Not correctly ignoring a node_modules subdirectory #1785
• Slow operations on very large repos #1841
• jj tracks ignored files #2051
• Initially-gitignored file continually reappears (with watchman) #2613
• jj status reports link in Fuchsia repo as removed #2878
• FR: Native submodules, submodule specific config #2919
• Support unignoring files/folders in .gitignore #3031
• Adding a gitignore after the fact is difficult to clean up #3203
• jj status stale cache of .gitignore #3336
• Weird corner-case issue with .gitignore in git init'ed repos #4222
• git submodules in colocated repos create weird phantom file creations/deletions #4349
• On a machine with many cores, snapshotting large repos is very CPU-intensive #4508
• Changing gitignore doesn't changing working commit #4686
• jj ignoring trailing slash in .gitignore #4718
• Checking out commit with submodule update fails #4825
• Git submodules not ignored #5005
• Submodule regression from 0.22 -> 0.23 #5013
• gitignore incompatibility with *~ #5059
• FR: Make jj file untrack work with files that are not ignored #5225
• Git submodule gets deleted and cannot be restored #5246
• FR: Improve/expand the 1MB file auto-snaphot heuristic #5278

[ilyagr]

Do you just mean running git -C git_repo_path status in a non-colocated repo? In other words, would this be any different from running git status in a colocated repo?

It's not yet implemented, but I think it'd be nice if something like git -C $(jj root --git) status worked.

[arxanas]

In other words, would this be any different from running git status in a colocated repo?

If jj status (and working tree snapshotting in general) itself calls git status, then we get e.g. Git LFS support for free, and jj status won't report logically-unchanged files as dirty.

[arxanas]

would this be any different from running git status in a colocated repo?

Well, I guess it depends on what you mean by "running git status".

• git status doesn't really have side effects, so I assumed you meant "would the output of jj status be logically different from the output of git status", for which the answer would be "it was before, and after this, it wouldn't be (in many cases)".
• But you might have meant something about the implementation of jj status calling into ("running") git status and whether it would be substantially complicated beyond just running git status. You have to pass --porcelain=v2 or something, but it should be straightforward.
  • The main problem would be handling tree entries that have hashes for objects that jj hasn't seen and that it can't procure itself, such as new Git LFS files. We'd have to warn or error on that case and require that the user resolve the issue using Git.

[ilyagr]

I think I see what you mean now. I read the part of your original post about the fsmonitor, but then forgot about it.

What do you mean by "Would handle submodules (if unchanged)."? I'm guessing you want to have a line in jj status that mentions that a submodule was changed and tells the user to use Git to work with it? That would be nice :).

Some disorganized first thoughts:

libgit2 has some support for this: https://libgit2.org/libgit2/#v0.20.0/group/status. I'm not sure whether it can handle LFS or submodules, and it wouldn't help with custom fsmonitors.

I think we'd still want to do a snapshot on jj status, but we could do it after running git status. The snapshot would go into the descendant of HEAD, so git status would still report the same diffs after it happens. Or perhaps we can call this jj status --no-snapshot.

One situation where this wouldn't work at all (I think) is if the repo has conflicted files. I suppose we could fall back to normal jj behavior in that case.

The problem of be handling tree entries that have hashes for objects that jj hasn't seen will have to be handled eventually, it'd be needed for any kind of large file support.

Currently, jj status shows the output of jj diff -s. Would we also change how diff works?

[arxanas]

Update: I had some work in progress here https://github.com/arxanas/jj/commits/5cc4c396ac01cbc1c55f6295cfecf21d852cb747/, which was successfully snapshotting based on git status, but it was producing incorrect incremental snapshots in some cases, and I hadn't yet debugged what the problem was. (Like not showing changes when you switched between commits or made a new change.)