Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

steam cache keeps missing some chunks #96

Open
Francesco149 opened this issue Dec 10, 2019 · 18 comments
Open

steam cache keeps missing some chunks #96

Francesco149 opened this issue Dec 10, 2019 · 18 comments

Comments

@Francesco149
Copy link

Francesco149 commented Dec 10, 2019 •
edited
Loading

Describe the issue you are having

I was testing out the steam cache and it seems that even redownloading the same game multiple times, the cache keeps reporting a lot of missed chunk and this reflects in the slow download speed. in errors.log there doesn't seem anything relevant other than a handful of failed requests unrelated to the chunks I'm seeing missing

the steam cache folder keeps growing after every download by the way so it's definitely caching new stuff

How are you running the container(s)?

docker run --rm \
  --name cache-steam \
  -v /memevault/cache/steam/data:/data/cache \
  -v /memevault/cache/steam/logs:/data/logs \
  -p 192.168.1.24:80:80 \
  lancachenet/generic:latest

DNS Configuration

docker run --rm \
  --name lancache-dns \
  -p 192.168.1.2:53:53/udp \
  STEAMCACHE_IP=192.168.1.24 \
  lancachenet/lancache-dns:latest

Output of container(s)

example of the same chunk missing multiple times

$ grep 773d10050d99b2544665873ec2125b3bf273e8b2 steam/logs/access.log
192.168.1.2 / - - - [10/Dec/2019:03:13:07 +0000] "GET /cache64-fra1.steamcontent.com/depot/70403/chunk/773d10050d99b2544665873ec2125b3bf273e8b2 HTTP/1.1" 302 0 "-" "Valve/Steam HTTP Client 1.0" "MISS" "ctr-00075-eu-it-center.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:13:15 +0000] "GET /cache64-fra1.steamcontent.com/depot/70403/chunk/773d10050d99b2544665873ec2125b3bf273e8b2 HTTP/1.1" 200 999232 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi001-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:20:55 +0000] "GET /cache64-fra1.steamcontent.com/depot/70403/chunk/773d10050d99b2544665873ec2125b3bf273e8b2 HTTP/1.1" 200 999232 "-" "Valve/Steam HTTP Client 1.0" "HIT" "ctr-00075-eu-it-center.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:26:02 +0000] "GET /cache64-fra1.steamcontent.com/depot/70403/chunk/773d10050d99b2544665873ec2125b3bf273e8b2 HTTP/1.1" 200 999232 "-" "Valve/Steam HTTP Client 1.0" "HIT" "ctr-00075-eu-it-center.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:32:24 +0000] "GET /cache44-fra1.steamcontent.com/depot/70403/chunk/773d10050d99b2544665873ec2125b3bf273e8b2 HTTP/1.1" 200 999232 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
@Francesco149
Copy link
Author

it seems like it wants to redownload chunks when they are requested from different cdn's

$ grep e0b46aed321a384cbd314b01cb2e87303bf704c8 steam/logs/access.log
192.168.1.2 / - - - [10/Dec/2019:03:12:04 +0000] "GET /depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache46-fra1.steamcontent.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:20:38 +0000] "GET /cache65-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi001-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:25:15 +0000] "GET /cache68-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi001-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:31:34 +0000] "GET /cache65-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "HIT" "qn-10075-fi001-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:46:03 +0000] "GET /cache69-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:53:49 +0000] "GET /cache59-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"

@Francesco149
Copy link
Author

just tested on the monolithic container, same behaviour

@VibroAxe
Copy link
Member

VibroAxe commented Dec 10, 2019 •
edited
Loading

Something weird going on here, steam Appears to be doing a get for CDN/depot instead of the usual /depot

Could you upload your entire access.log somewhere (gist.github.com maybe) and tell us which game you were downloading this on

@MathewBurnett
Copy link
Member

[steam] 10.81.0.19 / - - - [10/Dec/2019:04:45:44 +0000] "GET /depot/271241/patch/3951236313252844930/7962357758666437985 HTTP/1.1" 200 187632 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache16-lhr1.steamcontent.com" "-"

As a reference this is the sort of thing we usually see. So we are attempting to narrow down. If it is a particular game/dept/country or cdn node for example

@MathewBurnett
Copy link
Member

depot 70401 is Recettear: An Item Shop's Tale
https://steamdb.info/app/70400/

@Francesco149
Copy link
Author

it does the same thing for other games. I'm in Italy. this is my access log with the monolithic container
https://gist.github.com/1dcbb2fec321638176a5727133befc4b

@Francesco149
Copy link
Author

this is the linux steam client by the way

@MathewBurnett
Copy link
Member

can you try changing the download region in the steam settings -> downloads -> download region for me?

@Francesco149
Copy link
Author

tried Germany, UK London and US Miami and it looks the same. see the bottom of this access log

https://gist.github.com/e8e82a467e8280857afee67dede4f20d

@Francesco149
Copy link
Author

I also tested the windows steam client today, with the same result. other services seem to cache fine (I tested blizzard launcher and it served overwatch fine). valve must've changed something in their cdn infrastructure. I'm guessing we're gonna have to have some way to strip that leading cache path or something

@MathewBurnett
Copy link
Member

Thanks for confirming that the region choice changes nothing. It looks like your part of the world resolves to a cqloud cdn. We are attempting to understand why they behave differently. Its possible its a misconfig at the value end

@andrewt81
Copy link

andrewt81 commented Jan 7, 2020 •
edited
Loading

[EDIT] Added more infos

Same here, i'm from italy and i experience exactly the same problem with your monolithic docker. I'm not sure if i had to open another issue on monolithic or if i can continue here.

Docker was freshly installed so it's the latest available (yesterday)

My configuration:
docker run --restart unless-stopped --name lancache --detach -v /data/cache:/data/cache -v /data/logs:/data/logs -p 192.168.246.180:80:80 lancachenet/monolithic:latest

docker run --restart unless-stopped --name lancache-sniproxy -p 192.168.246.180:443:443 lancachenet/sniproxy:latest

DNS redirected from pfsensezeropingheroes/lancache-dns-pfsense

I changed region to UK without any change.

speeds are not consistent download to download

First download at DSL speed
second download, begin at gigabit speed but then at DSL speed with some spikes
third download, begin at DSL speed with Spikes at Gigabit speed

some examples:

[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:05 +0000] "GET /cache34-lhr1.steamcontent.com/depot/230291/chunk/cb1f19fa78e552a58008de7c799d72bf1fa6817a HTTP/1.1" 200 458320 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:05 +0000] "GET /cache34-lhr1.steamcontent.com/depot/230291/chunk/cb1f19fa78e552a58008de7c799d72bf1fa6817a HTTP/1.1" 200 458320 "-" "Valve/Steam HTTP Client 1.0" "HIT" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:05 +0000] "GET /cache29-lhr1.steamcontent.com/depot/230291/chunk/47e04b1e588fcc832c98132a45f7037d4dbfcaf8 HTTP/1.1" 200 213264 "-" "Valve/Steam HTTP Client 1.0" "HIT" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:05 +0000] "GET /cache29-lhr1.steamcontent.com/depot/230291/chunk/5c574103bd3bcf27f18ce685ef1e0ac4252bf731 HTTP/1.1" 200 18400 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:06 +0000] "GET /cache8-lhr1.steamcontent.com/depot/230291/chunk/70ab0794caf6878f6daf4f5a10a53be2b5b8853a HTTP/1.1" 200 280912 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"

Even if the game is always the same, the cache continues to grow download after download.

@mintopia
Copy link
Member

mintopia commented Jan 7, 2020

I've mentioned this issue to Valve (with regards to the hostname being in the path, which is preventing caching) and had this response:

Regarding the “host name in the path” requests – that is related to Open Cache proxies. ISPs are setting these up to cache Steam download traffic; we direct clients to use them when they match the IP filter list supplied by their API.

The good news is, there's an update in the Steam Beta client which, when rolled out to non-beta, with an update to the cache domains list, should resolve this and some of the other Valve issues.

@Lepidopterist
Copy link
Member

Further to this - The changes to the Steam client are now in the Live client.

If the relevant parties in this issue could retest with the latest lancachenet/monolithic container, the latest lancachenet/dns container, and the latest uklans/cache-domains list, and see if this issue has been resolved?

Many thanks

@andrewt81
Copy link

I'll test in the next days.. now little holiday :)

This issue is about generic, are the same updates working also in monolithic?

@Lepidopterist
Copy link
Member

The Valve client change will work with both Monolithic and Generic, so test with whichever you are comfortable using.

@Francesco149
Copy link
Author

seems to work fine now, at least on the windows client. I had to download the game twice to fully warm up the cache (maybe it was using some different chunks the 2nd time around), but it's definitely hitting all the chunks it has seen once

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mintopia @Francesco149 @VibroAxe @MathewBurnett @andrewt81 @Lepidopterist and others