cabal update does not bring revisions of metadata into effect.

[kindaro]

Describe the bug
Take a look at the revision here. Its aim is to make the use of this version impossible, because it is broken: pcapriotti/optparse-applicative#475. However, I can still build stuff that depends on this broken version even after I run cabal update. It seems metadata is not updated in my local copy of this package that Cabal downloaded before the revision was made.

I do confirm that fetching this version anew with cabal get and trying to build that freshly fetched package results in dependency resolution error, as expected.

To Reproduce

1. Upload a package x of version v to Hackage.
2. Create a local package y that depends on x of version v and only of version v.
3. See that y builds.
4. Make a revision to the metadata of the package x of version v on Hackage such that it can never be resolved.
5. Run cabal update.
6. See that y still builds, even though it now depends on an unresolvable version v of x.

Expected behavior
y should not build, because the revised metadata of its dependency x make it unresolvable.

System information

% cabal --version
cabal-install version 3.8.1.0
compiled using version 3.8.1.0 of the Cabal library
% uname
Linux

[ffaf1]

I quickly tried to reproduce this, but it seems I am failing. What am I doing wrong?

f@mkiii:/tmp/prova$ cat prova.cabal
cabal-version:   3.0
name:            prova
version:         0.1.0.0
license:         NONE
author:          Francesco Ariis
maintainer:      fa-ml@ariis.it
build-type:      Simple
extra-doc-files: CHANGELOG.md

common warnings
    ghc-options: -Wall

executable prova
    import:           warnings
    main-is:          Main.hs
    build-depends:    base,
                      optparse-applicative == 0.18.0.0
    hs-source-dirs:   app
    default-language: Haskell2010
f@mkiii:/tmp/prova$ cabal build
Warning: this is a debug build of cabal-install with assertions enabled.
Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: prova-0.1.0.0 (user goal)
[__1] next goal: optparse-applicative (dependency of prova)
[__1] rejecting: optparse-applicative-0.18.1.0 (conflict: prova =>
optparse-applicative==0.18.0.0)
[__1] skipping: optparse-applicative-0.17.1.0, optparse-applicative-0.17.0.0,
optparse-applicative-0.16.1.0, optparse-applicative-0.16.0.0,
optparse-applicative-0.15.1.0, optparse-applicative-0.15.0.0,
optparse-applicative-0.14.3.0, optparse-applicative-0.14.2.0,
optparse-applicative-0.14.1.0, optparse-applicative-0.14.0.0,
optparse-applicative-0.13.2.0, optparse-applicative-0.13.1.0,
optparse-applicative-0.13.0.0, optparse-applicative-0.12.1.0,
optparse-applicative-0.12.0.0, optparse-applicative-0.11.0.2,
optparse-applicative-0.11.0.1, optparse-applicative-0.11.0,
optparse-applicative-0.10.0, optparse-applicative-0.9.1.1,
optparse-applicative-0.9.1, optparse-applicative-0.9.0,
optparse-applicative-0.8.1, optparse-applicative-0.8.0.1,
optparse-applicative-0.8.0, optparse-applicative-0.7.0.2,
optparse-applicative-0.7.0.1, optparse-applicative-0.7.0,
optparse-applicative-0.6.0, optparse-applicative-0.5.2.1,
optparse-applicative-0.5.2, optparse-applicative-0.5.1,
optparse-applicative-0.5.0, optparse-applicative-0.4.3,
optparse-applicative-0.4.2, optparse-applicative-0.4.1,
optparse-applicative-0.4.0, optparse-applicative-0.3.2,
optparse-applicative-0.3.1, optparse-applicative-0.3.0,
optparse-applicative-0.2.0, optparse-applicative-0.1.1,
optparse-applicative-0.1.0, optparse-applicative-0.0.1 (has the same
characteristics that caused the previous version to fail: excluded by
constraint '==0.18.0.0' from 'prova')
[__1] trying: optparse-applicative-0.18.0.0
[__2] next goal: base (dependency of prova)
[__2] rejecting: base-4.18.0.0/installed-4.18.0.0 (conflict:
optparse-applicative => base<0)
[__2] skipping: base-4.18.0.0, base-4.17.2.0, base-4.17.1.0, base-4.17.0.0,
base-4.16.4.0, base-4.16.3.0, base-4.16.2.0, base-4.16.1.0, base-4.16.0.0,
base-4.15.1.0, base-4.15.0.0, base-4.14.3.0, base-4.14.2.0, base-4.14.1.0,
base-4.14.0.0, base-4.13.0.0, base-4.12.0.0, base-4.11.1.0, base-4.11.0.0,
base-4.10.1.0, base-4.10.0.0, base-4.9.1.0, base-4.9.0.0, base-4.8.2.0,
base-4.8.1.0, base-4.8.0.0, base-4.7.0.2, base-4.7.0.1, base-4.7.0.0,
base-4.6.0.1, base-4.6.0.0, base-4.5.1.0, base-4.5.0.0, base-4.4.1.0,
base-4.4.0.0, base-4.3.1.0, base-4.3.0.0, base-4.2.0.2, base-4.2.0.1,
base-4.2.0.0, base-4.1.0.0, base-4.0.0.0, base-3.0.3.2, base-3.0.3.1 (has the
same characteristics that caused the previous version to fail: excluded by
constraint '<0' from 'optparse-applicative')
[__2] fail (backjumping, conflict set: base, optparse-applicative, prova)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: prova, optparse-applicative, base

[kindaro]

You did not have a local copy of optparse-applicative-0.18.0.0 that was downloaded before the metadata revision. You have to try it with a new package — maybe at your own local installation of Hackage. In other words, you skipped the step 3.

[fgaz]

Maybe it can be reproduced by messing with --index-state?

[andreabedini]

I cannot reproduce.

~/Scratchpad
❯ export CABAL_DIR=$(mktemp -d)

~/Scratchpad
❯ mkdir cabal-issue-9201

~/Scratchpad
❯ cd cabal-issue-9201

~/Scratchpad/cabal-issue-9201
❯ cabal init -m -n -d base,optparse-applicative==0.18.0.0
Warning: The package list for 'hackage.haskell.org' does not exist. Run 'cabal
update' to download it.
[Log] Using cabal specification: 3.0
[Warning] unknown license type, you must put a copy in LICENSE yourself.
[Log] Creating fresh file CHANGELOG.md...
[Log] Creating fresh directory ./app...
[Log] Creating fresh file app/Main.hs...
[Log] Creating fresh file cabal-issue9201.cabal...
[Warning] No synopsis given. You should edit the .cabal file and add one.
[Info] You may want to edit the .cabal file and add a Description field.

~/Scratchpad/cabal-issue-9201
❯ cabal update
Downloading the latest package list from hackage.haskell.org
Package list of hackage.haskell.org has been updated.
The index-state is set to 2023-08-24T11:38:29Z.

~/Scratchpad/cabal-issue-9201 23s
❯ cabal build --index-state=2023-08-24T00:43:22Z
Warning: Requested index-state 2023-08-24T00:43:22Z is newer than
'hackage.haskell.org'! Falling back to older state (2023-08-23T19:58:30Z).
Resolving dependencies...
Build profile: -w ghc-9.4.6 -O1
In order, the following will be built (use -v for more details):
 - colour-2.3.6 (lib) (requires download & build)
 - prettyprinter-1.7.1 (lib) (requires download & build)
 - transformers-compat-0.7.2 (lib) (requires download & build)
 - ansi-terminal-types-0.11.5 (lib) (requires download & build)
 - ansi-terminal-1.0 (lib) (requires download & build)
 - prettyprinter-ansi-terminal-1.1.3 (lib) (requires download & build)
 - optparse-applicative-0.18.0.0 (lib) (requires download & build)
 - cabal-issue9201-0.1.0.0 (exe:cabal-issue9201) (first run)
...
Starting     optparse-applicative-0.18.0.0 (lib)
Building     optparse-applicative-0.18.0.0 (lib)
Installing   optparse-applicative-0.18.0.0 (lib)
Completed    optparse-applicative-0.18.0.0 (lib)
Configuring executable 'cabal-issue9201' for cabal-issue9201-0.1.0.0..
Preprocessing executable 'cabal-issue9201' for cabal-issue9201-0.1.0.0..
Building executable 'cabal-issue9201' for cabal-issue9201-0.1.0.0..
[1 of 1] Compiling Main             ( app/Main.hs, /home/andrea/Scratchpad/cabal-issue-9201/dist-newstyle/build/x86_64-linux/ghc-9.4.6/cabal-issue9201-0.1.0.0/x/cabal-issue9201/build/cabal-issue9201/cabal-issue9201-tmp/Main.o )
[2 of 2] Linking /home/andrea/Scratchpad/cabal-issue-9201/dist-newstyle/build/x86_64-linux/ghc-9.4.6/cabal-issue9201-0.1.0.0/x/cabal-issue9201/build/cabal-issue9201/cabal-issue9201

~/Scratchpad/cabal-issue-9201 18s
❯ cabal build --index-state=2023-08-24T00:43:23Z
Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: cabal-issue9201-0.1.0.0 (user goal)
[__1] next goal: optparse-applicative (dependency of cabal-issue9201)
[__1] rejecting: optparse-applicative-0.18.1.0 (conflict: cabal-issue9201 =>
optparse-applicative==0.18.0.0)
...
[__1] trying: optparse-applicative-0.18.0.0
[__2] next goal: base (dependency of cabal-issue9201)
[__2] rejecting: base-4.17.2.0/installed-4.17.2.0 (conflict:
optparse-applicative => base<0)
...
[__2] fail (backjumping, conflict set: base, cabal-issue9201,
optparse-applicative)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: cabal-issue9201, optparse-applicative,
base

[kindaro]

What if you try to follow the steps I offered?

I tried it out again and I can still reproduce the erratic behaviour.

• install and run hackage-server
• create two plausibly looking packages literally named x and y
• upload x
• make y depend on x
• build y
• revise the metadata of x such that it is unresolvable
• run cabal update
• build `y

— Reproduction successful.

A tip to speed you up:

repository my-hackage
  url: http://localhost:8080/

— Put this into $CABAL_DIR/config instead of whatever it says in the repository section so that it be working with your local Hackage server installation.

Perhaps that the failure I described cannot be reproduced by messing with --index-state tells us something about where the fault is? I think Cabal forgets to update some local cache because it does not see the metadata update as a significant change.

[andreabedini]

@kindaro I tried following your steps and put together this script:

https://gist.github.com/andreabedini/283ee0213f304050bcddae830933d0c1

With this script I cannot reproduce the issue.

Nevertheless the tip "to speed me up" gives away the issue you must have been facing:

repository my-hackage
  url: http://localhost:8080/

Without secure: True, this specifies a "legacy" repository which do not support revisions and/or index-state.

Take out the secure: True in my script and you will get the desired result.

Were you using an internal repository by any chance?

[andreabedini]

Please reopen if the problem is not fully resolved.

[kindaro]

Thank you Andrea @andreabedini. This is a beautifully written shell script. I did not run it — I believe you. The cause you pointed out is likely true. I see that the line secure: True is commented out in my Cabal configuration file.

repository hackage.haskell.org
  url: http://hackage.haskell.org/
  -- secure: True
  -- root-keys:
  -- key-threshold: 3

Is this the default situation? Perhaps I had commented it out some time ago — I do not remember. If so it was my mistake. Although…

The documentation says:

When interacting with hackage.haskell.org, Cabal always runs in secure mode with standard root keys, so it is not necessary to specify secure or root-keys. If no repositories are listed, Cabal will default to hackage.haskell.org.

— https://cabal.readthedocs.io/en/stable/config.html#using-secure-repositories

This assertion seems to be false. According to you, the issue can be seen locally exactly when Cabal runs outside of secure mode. According to the documentation I quote, Cabal always runs in secure mode when interacting with hackage.haskell.org. But I stumbled upon this error for the first time when interacting with hackage.haskell.org! I see a contradiction here.

If, contrary to the documentation, Cabal runs outside of secure mode by default and without exceptions, it seems there is a dangerous pitfall. Security should be something you must opt out of, not opt in. For example, Elasticsearch recently added security by default even though this change makes it incompatible with some of the installations existing at that time.

In Elastic 8.0, security is now enabled by default for self-managed clusters.

— https://www.elastic.co/blog/whats-new-elastic-8-0-0

I should also say that the implications of secure mode are not clear to me. My Cabal installation does accept the flag --index-state, so it seems this is supported. I see nothing in the documentation about revisions not being supported when Cabal runs outside of secure mode. Perhaps there is room to make the documentation better here.

I fear the problem is not fully resolved yet… I cannot open this issue again because there is no such button on offer. Perhaps you could check the settings of this repository and allow this action.

[andreabedini]

Thank you Andrea @andreabedini. This is a beautifully written shell script.

Thank you, it did take a good few hours of my time, I am glad it wasn't time wasted.

repository hackage.haskell.org
url: http://hackage.haskell.org/
-- secure: True
-- root-keys:
-- key-threshold: 3

Is this the default situation? Perhaps I had commented it out some time ago — I do not remember.

Nah, this is the default configuration as initialised by cabal. TBH I have not idea how we write out the default values as commented options 🤷

When interacting with hackage.haskell.org, Cabal always runs in secure mode with standard root keys, so it is not necessary to specify secure or root-keys. If no repositories are listed, Cabal will default to hackage.haskell.org.
— https://cabal.readthedocs.io/en/stable/config.html#using-secure-repositories

This assertion seems to be false. According to you, the issue can be seen locally exactly when Cabal runs outside of secure mode.

I agree with your conclusion. One because you can explicitly set secure: False and cabal will not use the "secure mode". Two because it looks like the configuration as written above does not correspond to what is actually used 😱

Proof:

❯ export CABAL_DIR=$(mktemp -d)
❯ cabal user-config init
Writing default configuration to /tmp/tmp.MxYwm3NJor/config
❯ grep -A4 repository $CABAL_DIR/config
repository hackage.haskell.org
  url: http://hackage.haskell.org/
  -- secure: True
  -- root-keys:
  -- key-threshold: 3
❯ cat cabal.project
repository hackage.haskell.org
  url: http://hackage.haskell.org/
  -- secure: True
  -- root-keys:
  -- key-threshold: 3

-- I need to have something to target
extra-packages: base
❯ cabal build --dry-run base
Warning: There are no packages or optional-packages in the project
Warning: The package list for 'hackage.haskell.org' does not exist. Run 'cabal
update' to download it.
Warning: The package list for 'hackage.haskell.org' does not exist. Run 'cabal
update' to download it.
Resolving dependencies...
Up to date

If I then inspect the resulting project configuration¹, I see

a =
  [ RemoteRepo
      { remoteRepoName = RepoName {unRepoName = "hackage.haskell.org"},
        remoteRepoURI = http://hackage.haskell.org /,
        remoteRepoSecure = Just True,
        remoteRepoRootKeys = ["fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0", "1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42", "2c6c3627bd6c982990239487f1abd02e08a02e6cf16edb105a8012d444d870c3", "0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d", "51f0161b906011b52c6613376b1ae937670da69322113a246a09f807c62f6921"],
        remoteRepoKeyThreshold = 3,
        remoteRepoShouldTryHttps = True
      },
    RemoteRepo
      { remoteRepoName = RepoName {unRepoName = "hackage.haskell.org"},
        remoteRepoURI = http://hackage.haskell.org /,
        remoteRepoSecure = Nothing,
        remoteRepoRootKeys = [],
        remoteRepoKeyThreshold = 0,
        remoteRepoShouldTryHttps = False
      }
  ]

where the first comes from the global configuration and the second comes from the project config. Notice that they are indentical in their textual representation.

I reckon all this should be revisited (documentation and logic). @gbaz are you aware of this?

 But I stumbled upon this error for the first time when interacting with hackage.haskell.org!

I understand but without a reproducible example there is nothing I can do.

I should also say that the implications of secure mode are not clear to me. My Cabal installation does accept the flag --index-state, so it seems this is supported. I see nothing in the documentation about revisions not being supported when Cabal runs outside of secure mode. Perhaps there is room to make the documentation better here.

You are correct. non secure-secure mode is often referred to as "legacy mode" in the codebase. How different kind of repositories are configured and handled in the codebase is another thing I would like to see refactored but alas all my attempts at wrangling it have failed so far.

I fear the problem is not fully resolved yet… I cannot open this issue again because there is no such button on offer. Perhaps you could check the settings of this repository and allow this action.

I don't have permission to change the settings but I can reopen the issue for you.

Footnotes

1. I made a tool for this but you can hack your own following this logic. ↩