Download bank with user-agent check

[braffes]

Hello everyone,

I am facing an issue with cloudflare ban because of my user agent.

To understand what is going on, I set the verbose mode about the request
self.crl.setopt(pycurl.VERBOSE, 1)

[...]
2024-11-11 23:35:02,897 INFO  [root][MainThread] Workflow:wf_release:DownloadSession:25740d7d-d436-4557-9112-40d1b177db2d                                                               
*   Trying 172.67.144.214...                                                                                                                                                            
* TCP_NODELAY set                                                                                                                                                                       
* Connected to bfvd.steineggerlab.workers.dev (172.67.144.214) port 443 (#0)                                                                                                            
* ALPN, offering http/1.1                                                                                                                                                               
* successfully set certificate verify locations:                                                                                                                                        
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt                                                                                                                                            
  CApath: none                                                                                                                                                                          
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384                                                                                                                                 
* ALPN, server accepted to use http/1.1                                                                                                                                                 
* Server certificate:                                                                                                                                                                   
*  subject: CN=steineggerlab.workers.dev                                                                                                                                                
*  start date: Sep 26 22:14:42 2024 GMT                                                                                                                                                 
*  expire date: Dec 25 22:14:41 2024 GMT                                                                                                                                                
*  subjectAltName: host "bfvd.steineggerlab.workers.dev" matched cert's "*.steineggerlab.workers.dev"                                                                                   
*  issuer: C=US; O=Google Trust Services; CN=WE1                                                                                                                                        
*  SSL certificate verify ok.                                                                                                                                                           
> GET /bfvd.version HTTP/1.1                                                                                                                                                            
Host: bfvd.steineggerlab.workers.dev
User-Agent: PycURL/7.45.1 libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0                 
Accept: */*

< HTTP/1.1 403 Forbidden
< Date: Mon, 11 Nov 2024 22:35:02 GMT
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 16
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< Referrer-Policy: same-origin
< Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0                                                                                    
< Expires: Thu, 01 Jan 1970 00:00:01 GMT
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=glIJPclq1Q1fOWRLa3fd6qa0MEv6J8msgiABtz86fnEjniyMhgTZdVYEXy6aR820rHEEwWOaZFar7sR3I3hEx10aJEBFMmuqbTk7IXa$vTvM4uea8FciTOHYp3r2e1pn%2FCwg7cE1DARpWh9kQSjX9d0%3D"}],"group":"cf-nel","max_age":604800}
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< Server: cloudflare
< CF-RAY: 8e11c0cf3f17017d-CDG
< alt-svc: h3=":443"; ma=86400
< server-timing: cfL4;desc="?proto=TCP&rtt=1994&sent=4&recv=7&lost=0&retrans=0&sent_bytes=2880&recv_bytes=868&delivery_rate=1387637&cwnd=252&unsent_bytes=0&cid=9ab9120bf167f7a0&ts=19&x
=0"
<
* Connection #0 to host bfvd.steineggerlab.workers.dev left intact
2024-11-11 23:35:02,928 ERROR [root][MainThread] Error while listing https://bfvd.steineggerlab.workers.dev/bfvd.version - 403                                                          
2024-11-11 23:35:02,928 ERROR [root][MainThread] Error while listing https://bfvd.steineggerlab.workers.dev/bfvd.version - Error while listing https://bfvd.steineggerlab.workers.dev/bf
vd.version - 403                                                                                                                                                                        
2024-11-11 23:35:02,928 ERROR [root][MainThread] b'error code: 1010'   

And when I put manually a custom agent like that self.crl.setopt(pycurl.USERAGENT, 'curl/7.45.0'), it works:

2024-11-11 23:34:32,447 INFO  [root][MainThread] Workflow:wf_release:DownloadSession:0a0bb372-0a90-485c-8a34-0e169d575fba                                                               
*   Trying 172.67.144.214...                                                                                                                                                            
* TCP_NODELAY set                                                                                                                                                                       
* Connected to bfvd.steineggerlab.workers.dev (172.67.144.214) port 443 (#0)                                                                                                            
* ALPN, offering http/1.1                                                                                                                                                               
* successfully set certificate verify locations:                                                                                                                                        
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt                                                                                                                                            
  CApath: none                                                                                                                                                                          
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384                                                                                                                                 
* ALPN, server accepted to use http/1.1                                                                                                                                                 
* Server certificate:                                                                                                                                                                   
*  subject: CN=steineggerlab.workers.dev                                                                                                                                                
*  start date: Sep 26 22:14:42 2024 GMT                                                                                                                                                 
*  expire date: Dec 25 22:14:41 2024 GMT                                                                                                                                                
*  subjectAltName: host "bfvd.steineggerlab.workers.dev" matched cert's "*.steineggerlab.workers.dev"                                                                                   
*  issuer: C=US; O=Google Trust Services; CN=WE1                                                                                                                                        
*  SSL certificate verify ok.                                                                                                                                                           
> GET /bfvd.version HTTP/1.1                                                                                                                                                            
Host: bfvd.steineggerlab.workers.dev                                                                                                                                                    
User-Agent: curl/7.45.0                                                                                                                                                                 
Accept: */*                                                                                                                                                                             
                                                                                                                                                                                        
< HTTP/1.1 200 OK                                                                                                                                                                       
< Date: Mon, 11 Nov 2024 22:34:33 GMT                                                                                                                                                   
< Content-Length: 8                                                                                                                                                                     
< Connection: keep-alive                                                                                                                                                                
< ETag: "772920eddf4d6ac2912dc90c1b68c847"                                                                                                                                              
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=S%2FoUFWJOA2dvVuTTXV3nvXSEKO0iv9rX8BRB4haRl%2BVjGh2FzI2DEOglV%2Fhf918zjqIeJcQfB3pXcrO7RXmKoGtDpuIyBImtA7
TcfWmUObrTqTuUqi8OKPvXr1xP15ShrAZKie7xNVXu%2FHY8t0pdKO8%3D"}],"group":"cf-nel","max_age":604800}                                                                                        
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}                                                                                                                     
< Server: cloudflare                                                                                                                                                                    
< CF-RAY: 8e11c010ede900d2-CDG                                                                                                                                                          
< alt-svc: h3=":443"; ma=86400                                                                                                                                                          
< server-timing: cfL4;desc="?proto=TCP&rtt=1924&sent=4&recv=7&lost=0&retrans=0&sent_bytes=2880&recv_bytes=725&delivery_rate=1435084&cwnd=252&unsent_bytes=0&cid=6b831ae3bd092756&ts=1204
&x=0"                                                                                                                                                                                   
<                                                                                                                                                                                       
* Connection #0 to host bfvd.steineggerlab.workers.dev left intact  

I am wondering if it was possible to set a custom user-agent during the download process?

Thanks for your help.