Spring Boot 3.4/Spring Cloud Dependencies 2024/OpenFeign 4.2 Upgrade Returns 403s For Every Call

[stephenmontgomery]

Describe the bug
Updating from Spring Boot 3.3.6 to Spring Boot 3.4, has lead to Feign returning 403/FORBIDDEN for every Feign call, for every FeignClient call.

Spring Boot 3.4.0
Spring Cloud Dependencies: 2024.0.0
Spring Cloud OpenFeign: 4.2.0
Apache Http Client5: 5.13.5

Rolling back to previous SB 3.3.6 version:
Spring Boot 3.3.6
Spring Cloud Dependencies: 2023.0.4
Spring Cloud OpenFeign: 4.1.4
Apache Http Client5: 5.3.1

gives:

No change to the component being called by Feign - it's still deployed/processing/unchanged. There are no error messages/exceptions in either component ie either the caller or callee - there is nothing to suggest any issue.

application.yml:

spring:
  cloud:
    openfeign:
      compression:
        request:
          enabled: true
        response:
          enabled: true

Has this been seen before? Any further troubleshooting steps I can do?

Thanks.

[OlgaMaciaszek]

Hello, @stephenmontgomery, thanks for reporting the issue. Please provide a minimal, complete, verifiable example that reproduces the issue.

[stephenmontgomery]

Hi @OlgaMaciaszek,
Yeah can't do that, unfortunately - it's only reproducible in our deployed environments - k8s with istio + envoy. Works fine with the same components deployed in my local docker compose.

Just tried https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.4-Release-Notes#apache-http-components-and-envoy without any luck.

[OlgaMaciaszek]

Hi @stephenmontgomery, there's only a single issue that comes to mind at this point that could be related in terms of changing the way the request is created: #1070 - you can see if those upstream changes are not affecting you and change the prop if they are. If this is not the case, you may want to enable full logging on the client side in SC OF (https://docs.spring.io/spring-cloud-openfeign/reference/spring-cloud-openfeign.html#spring-cloud-feign-overriding-defaults) on both versions and compare the requests to see what's changed.

[stephenmontgomery]

Hi @OlgaMaciaszek,
We do have that configurer.setUseTrailingSlashMatch(true) SB 3 workaround littered everywhere so prob not it but I'll double-check. There def doesn't seem to be any trailing slash in the Feign logs above.

Bit further on here though, with some of our Operations team's input - they found something similar with istio/istio#53239 - outlines the 403 behaviour even though using http and not https. As mentioned above, I did try https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.4-Release-Notes#apache-http-components-and-envoy - which should have workedaround this issue but it had no effect so I'm wondering if the default config didn't take.

I'm also considering downgrading the HC5 version. As per spring-projects/spring-boot#43139, this TLS stuff came in HC5 5.4. As mentioned above, the last SB 3.3.6 version used HC5 5.3.1

[OlgaMaciaszek]

I don't have a better suggestion than for you to compare the full requests on both versions. Once you identify the issue (i.e. what has changed and is breaking you, we can take a look at how to resolve it).

[stephenmontgomery]

Sorry @OlgaMaciaszek - I thought I have given the full Feign request logging. Is there additional debug logging I can add?

[OlgaMaciaszek]

I don't see any information about headers, for example. Some other request details might be missing. Also, we need to have the full log from both versions (the working one and then not working one) to compare for differences. Please make sure to add a Logger.Level bean in your Feign client configuration to return Logger.Level.FULL (as described in the linked doc) and make sure to set the logging.level for the package where the Feign Client is to DEBUG. Then send provide full request logs from the working and not working requests. Please provide them in text/log form and not as screenshots.

[stephenmontgomery]

Ok @OlgaMaciaszek worth checking but I confirm FULL feign logging was enabled and headers are seen (request + response) in the screenshots above. Can't give u actual text because accessible from DataDog and I'll have to search over again.

logging:
  level:
    root: INFO
    com.zzzzzzz: DEBUG
    com.zzzzzzz.zzzzz.api.client: DEBUG

spring:
  cloud:
    openfeign:
      client:
        config:
          default:
            logger-level: FULL

Anyho just making sure I'm not missing any extra diagnostic debug.

[OlgaMaciaszek]

That should be fine. If it takes effect, you should be easily able to spot it in the logs, as it's quite verbose.

[jaccarte]

Also seeing this behaviour. Having enabled logging in the HTTP client it looks like the suggested fix (to disable the protocol upgrade) isn't having the desired effect. The output is the same with or without the fix applied:

Strangely when we upgraded the HTTP client and applied a similar fix in the previous Spring Boot it worked fine.

[OlgaMaciaszek]

@jaccarte Could you please do this: #1141 (comment) and provide request info for comparison?

[jaccarte]

I did FULL request logging with and without the fix and it was exactly the same (no mention of the connection upgrade), hence why I enabled logging in the relevant hc5 class which deals with the actual connection upgrade. It showed the logs above even when I had supposedly disabled the automatic connection upgrade (those logs should only appear when the feature is enabled).

Anyhow I found a working solution by adding the following to my configuration. I guess the other fix isn't touching the Feign configuration for some reason (or maybe it's not intended to?).

@Bean
public HttpClient5FeignConfiguration.HttpClientBuilderCustomizer httpClientBuilder() {
  return (httpClientBuilder) -> {
    var rcBuilder = RequestConfig.custom();
    rcBuilder.setProtocolUpgradeEnabled(false);
    httpClientBuilder.setDefaultRequestConfig(rcBuilder.build());
  };
}

[OlgaMaciaszek]

@jaccarte thanks for the update and I'm happy there's a workaround.

when I had supposedly disabled the automatic connection upgrade

• can you please let me know in which way you set it up in your app? Could you also please provide a minimal, complete, verifiable example that reproduces the issue?

[OlgaMaciaszek]

I see now this is related to https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.4-Release-Notes#apache-http-components-and-envoy. Will look into creating a better way of handling this.

[jaccarte]

FYI the issue is can be easily reproducible whenever a plain non-encrypted HTTP call is made using a Feign client that hits Istio. Istio, instead of ignoring or upgrading the request to upgrade the connection to be encrypted, simply refuses the request and returns 403.