Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redump Serial vs Internal Serial #59

Open
kikmon opened this issue Jul 27, 2022 · 24 comments
Open

Redump Serial vs Internal Serial #59

kikmon opened this issue Jul 27, 2022 · 24 comments

Comments

@kikmon
Copy link

kikmon commented Jul 27, 2022

Hi, I see this topic has been discussed some time ago, but I had a few questions.
(Trying to fix some PSX games not scanning properly :) )
If I understand properly, Redump has 2 kind of serials.
The Internal, that is really in the disk data (and I think this is the one Retroach will find), and the Serial that appears on the disk labels / box.
If the internal has the same value as the regular one, internal is just omitted by Redump.
So to fix the serial scanning of these games that have an internal serial, we need the information in the RDB, and for, that we need to get from Redump's dat.
Is there a way to have the internal serial in the Redump dat ?
I see the tool calls http://redump.org/datfile/psx/serial,version to get the serial. Is there a URL to also get the internal one ?
If we can get this info in the rdb, then the scanning logic of RA could be altered to take the 'internal serial' into account if present.
@RobLoach
Copy link
Owner

All data that's grabbed from redump is build into .dat files over at:
https://github.com/libretro/libretro-database/tree/master/metadat/redump

This includes the serial number. For example...


game (
	name "Crazy Taxi (USA)"
	description "Crazy Taxi (USA)"
	region "USA"
	serial "51035"
	rom ( name "Crazy Taxi (USA) (Track 1).bin" size 1058400 crc F8BB5B3C md5 BE87B59616171C57D9FD252ACDDA806B sha1 CF38F6C262F304729870F443FCE69FFD182214E1 serial "51035" )
)

@kikmon
Copy link
Author

kikmon commented Jul 27, 2022 •
edited
Loading

Thanks for the followup.
Yes, but there's only one serial in this example, and the 'serial vs internal serial' plagues Multi Disc games mostly.

Let's take an example that RA is unable to scan: PS1 game Alive (Japan) (Disk 2)
On Redump, (http://redump.org/disc/22867/) the serial is 'SLPS 01528' (printed on Disk Label) and Internal Serial is 'SLPS-01527' (the one found in the track data)
The dat file you linked shows 2 entries for Alive (Japan) (Disk 2), featuring Serial SLPS 01528 and SLPS 01528-1, both derived from the regular Serial, but there is no trace of the internal serial (SLPS-01527)
When RA extracts the serial from the disk, it finds ... SLPS-01527, and since it's a multidisc game, as a good soldier, it adds -1, and then performs a lookup in the database for SLPS-01527-1, which is unknown in the DB

For the scanning to work, we'd need some knowledge of this 'Internal Serial'. It's not present in the Redump dat we're downloading, hence my question about the existence of a Redump Download URL that can return both serials. :)

@RobLoach
Copy link
Owner

Ah, I understand. That makes sense, thanks a lot. Curious about this as well. RetroArch requires one-rom to one-entry, so we would somehow need to split the entries.

@kikmon
Copy link
Author

kikmon commented Jul 27, 2022

Yep, Once we know SLPS-01527 is the internal one, we can maybe build an entry for SLPS-01527-1
But in order to have the right data, with need Redump to cooperate.
I've looked at my other scan failures, and here's another funny one
http://redump.org/disc/32630/ (SLPM 86306 vs SLPS-01487)
It's a single disc game, but the serial on the label doesn't match the internal serial :)

@RobLoach
Copy link
Owner

@pkos may be interested in some of this too. He had been working on this repo: https://github.com/pkos/CRoSG which brings in a few additional serials.

Tested out building it into the .dat files over at #53 . Could use some thoughts and testing.

@kikmon
Copy link
Author

kikmon commented Jul 28, 2022 •
edited
Loading

Interesting.
If I read this well, it's mainly to add chd crc scanning ? (I'm not seeing the serials in the dat)
Do we have a privileged communication channel with redump to get the missing info ?

@kikmon kikmon mentioned this issue Jul 31, 2022
Open
@kikmon
Copy link
Author

kikmon commented Aug 11, 2022

Ok, got in touch with the kind people at Redump.
Internal Serials are not part of the database, only comments, so we can't get them in the dat. There's a chance to have them in the future, but not anytime soon, so we must find another way to recognize these games.

@pkos
Copy link

pkos commented Aug 11, 2022 via email

@pkos
Copy link

pkos commented Aug 11, 2022 via email

@pkos
Copy link

pkos commented Aug 11, 2022 via email

@kikmon
Copy link
Author

kikmon commented Aug 12, 2022

Thanks for joining :)
As far as I see, there are 2 issues mixing up here:

  • I've played a bit in the RA scanner recently to improve PSX scanning, and RA will only fallback on crc if a serial cannot be extracted. So if a serial is found (like in the case of internal serials different from the one on the box), RA will continue doing the DB lookup and fail. Maybe we should do the fallback on the crc if the whole lookup fails.
  • Then, we have an issue with the RDB, as they are built using Serial as the key, several entries are missing, and crc lookup will fail (A Ressha de Ikou 4 - Evolution (Japan) is a good example). if we build the rdb with crc key instead, I think it would work better, as some serial would be missed on the first lookup, but then crc will find them.

@pkos
Copy link

pkos commented Aug 12, 2022 via email

@pkos
Copy link

pkos commented Aug 12, 2022 via email

@pkos
Copy link

pkos commented Aug 12, 2022 via email

@pkos
Copy link

pkos commented Aug 12, 2022 via email

@kikmon
Copy link
Author

kikmon commented Aug 13, 2022 •
edited
Loading

Thanks for your feedback :)

Rob could we get a quick patch to make sure the overall RDBs have a latest update date stamp. Redump is much more active about dats.

I don't think this would help much since the ps1 rdb is still created with the parameter 'rom.Serial'.
When transforming the .dat into .rdb, some information is lost. Maybe going back to rom.crc for PS1 would yield overall better results

The log files can be enabled for data on your complete scan if you enable logging in RA.

Yeah, I've been running RA Scanning in Debug mode and added more traces to ease log analysis. I'm working on a subset of 4000 JP games at the moment. Takes a while to scan :)
Right now I'm investigating 11 that fail scanning

The people at redump didn't inform you of a url that can be used to get serials in their days my friend, we both have had some good conversations in VGPS. The URL format is http://redump.org/datfile/ss/serial,version
Try http://redump.org/datfile/PSX/CRC,serial,version

The url /serial,version is already used to generate the redump metadat. We have the serials already, that's not the problem. What we're missing are the 'Internal Serials' that are not part of Redump Database.
As mentioned earlier here's a good example http://redump.org/disc/32630/ Notice the difference between the 'Serial' and 'Internal Serial'

@RobLoach
Copy link
Owner

@kikmon What's the difference between Internal Serial and Serial?

@kikmon
Copy link
Author

kikmon commented Dec 20, 2023

Disclaimer, I'm mostly talking about PS1 games here :)
Serial is the Serial number that appears on the game cover, disc label, etc.
Internal Serial, is the Serial found in the data of the disc, the one Retroarch will find when scanning.
Usually, they are the same, but for some games, they can be different, preventing Retroarch from scanning the game properly.
Redump contributors started adding the Internal Serial information, but unfortunately, it is part of the Comments, and not in the database, meaning it's not part of the Dat.
Unfortunately, Redump team seemed to be unable to change the database structure for now, so this information, useful for scanning, can't be used as-si

@RobLoach
Copy link
Owner

Very cool. Seems a lot more useful than the Serial. Do you know the URL code to grab the Internal Serial data in the dat? For example...
http://redump.org/datfile/PSX/CRC,serial,version,internal_serial

or something?

@kikmon
Copy link
Author

kikmon commented Dec 20, 2023 •
edited
Loading

As far as I know, it's not possible right now.
There is no database field for the Internal Serial, as they are parts of the Comments, and I don't think we can request the comments to be included in the dat file
As mentioned earlier, this entry shows the issue :)
http://redump.org/disc/32630/
Internal and 'regular' serial are quite different, and Internal Serial is in the Comment section

@RobLoach
Copy link
Owner

That's unfortunate. We likely won't be able to include it then.

@kikmon
Copy link
Author

kikmon commented Dec 21, 2023

yeah, I don't think we can do it the 'clean way'
Now, we could scrape Redump website to extract the Internal Serial from the webpages directly, and patch the official dat file by adding a new field :)
Unfortunately, because of internal issues, I don't think Redump will be able to update it's DB scheme.
Unless some changes occur in Retroarch Scanning logic, these games with different internal serial will never be scanned properly

@RobLoach
Copy link
Owner

If you're up to manually do it, could add entries manually to the /dat folder https://github.com/libretro/libretro-database/tree/master/dat

@kikmon
Copy link
Author

kikmon commented Dec 26, 2023

Are you talking about https://github.com/libretro/libretro-database/blob/master/dat/ps1.idlst ?
Is this file combined with the official Redump DAT to create the final RDB ?
If that's the case, then yeah, I could probably add a few problematic Serials in this file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@RobLoach @pkos @kikmon