Temp #444
Open
Temp #444
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hmottestad
commented
Jan 22, 2024
Comment on lines
+39
to
+83
| public void setupRepo() throws IOException { | ||
| Path root = tempDir.newFolder().toPath(); | ||
| ClassLoader loader = getClass().getClassLoader(); | ||
| String filename = "2018_complete.nt"; | ||
|
|
||
| Path hdtstore = root.resolve("hdt-store"); | ||
| Path locationNative = root.resolve("native"); | ||
|
|
||
| Files.createDirectories(hdtstore); | ||
| Files.createDirectories(locationNative); | ||
|
|
||
| String indexName = "index.hdt"; | ||
|
|
||
| HDTOptions options = HDTOptions.of( | ||
| // disable the default index (to use the custom indexes) | ||
| HDTOptionsKeys.BITMAPTRIPLES_INDEX_NO_FOQ, true, | ||
| // set the custom indexes we want | ||
| HDTOptionsKeys.BITMAPTRIPLES_INDEX_OTHERS, "sop,ops,osp,pso,pos"); | ||
|
|
||
|
|
||
| try (HDT hdt = HDTManager.generateHDT(new Iterator<>() { | ||
| @Override | ||
| public boolean hasNext() { | ||
| return false; | ||
| } | ||
|
|
||
| @Override | ||
| public TripleString next() { | ||
| return null; | ||
| } | ||
| }, Utility.EXAMPLE_NAMESPACE, options, null)) { | ||
| hdt.saveToHDT(hdtstore.resolve(indexName).toAbsolutePath().toString(), null); | ||
| } catch (Error | RuntimeException e) { | ||
| throw e; | ||
| } catch (Exception e) { | ||
| throw new RuntimeException(e); | ||
| } | ||
|
|
||
| repository = CompiledSail.compiler().withEndpointFiles(new EndpointFiles(locationNative, hdtstore, indexName)) | ||
| .compileToSparqlRepository(); | ||
| try (InputStream is = new BufferedInputStream(Objects.requireNonNull(loader.getResourceAsStream(filename), | ||
| filename + " doesn't exist"))) { | ||
| repository.loadFile(is, filename); | ||
| } | ||
| } |
There was a problem hiding this comment.
Can you detail this a bit more. I loaded the dataset and I was querying one triple pattern and I see that the cardinality is non zero:
There was a problem hiding this comment.
Try adding a break point on that line and then debugging the TempTest test() I added.
There was a problem hiding this comment.
Don't forget to add the file with the data to this path: qendpoint-store/src/test/resources/2018_complete.nt
There was a problem hiding this comment.
It's a race condition because a merge was triggered in the middle of the file loading
There was a problem hiding this comment.
Do you know how I can wait for the merge to complete?
There was a problem hiding this comment.
Btw. In my PR eclipse-rdf4j/rdf4j#4879 I managed to get the QueryJoinOptimizer to work better when the statistics are inaccurately returning 0.0 by adding 5.0 to the cost of all operations and also penalising cartesian joins more heavily.
hmottestad
commented
Jan 22, 2024
| @@ -0,0 +1,123 @@ | |||
| Distinct (resultSizeActual=12, totalTimeActual=9.8s, selfTimeActual=0.048ms) | |||
There was a problem hiding this comment.
I managed to manually optimize the query and it runs in about 10 seconds on my laptop.
There was a problem hiding this comment.
ah, you mean, you still have the problem that the query planner is not woriking and you tried to reorder the triple patterns by hand only?
There was a problem hiding this comment.
Yeah. I managed to reorder the query by hand.
hmottestad
commented
Jan 23, 2024
Comment on lines
+124
to
+125
| # This line should be above the FILTER and BIND, but doing so causes the QueryJoinOptimizer to not optimize the query for some unknown reason | ||
| ?resultnotice epo:refersToRole ?buyerrole . |
There was a problem hiding this comment.
This line should be above the FILTER and BIND. When I put it above then the QueryJoinOptimizer doesn't manage to optimize the query at all. When I move it back down again like I've done now then it does end up triggering the QueryJoinOptimizer seemingly correctly, but it only seems to check the cardinality of this single statement pattern here and when it does it gets 0.0 as the cardinality.
If you run the test you should see the following printed:
Cardinality for StatementPattern Var (name=resultnotice) Var (name=_const_6aa9a9c_uri, value=http://data.europa.eu/a4g/ontology#refersToRole, anonymous) Var (name=buyerrole) is 0.0 (HDT: 0.0, Native: 0.0)
There was a problem hiding this comment.
I had a look at the QueryJoinOptimizer code and it doesn't seem to recurse down into the query tree. Essentially it will recurse down to the first Join and then optimize the join arguments that are either instance of Join or LeftJoin or StatementPattern. Removing the inner sub-select from the query "fixes" the QueryJoinOptimizer.
There was a problem hiding this comment.
I fixed the QueryJoinOptimizer eclipse-rdf4j/rdf4j#4879
|
|
|
|
|
I've merged the fix for the QueryJoinOptimizer and it's available from the snapshot repo. The query still needs a slight modification. RDF4J does not optimise the location of the BIND clause, so it is unable to move the My original theory that RDF4J can't optimise sub-selects was wrong. Queries with a single sub-select can be optimised, but not if the non-sub-select parts contain a BIND clause. They can also not be optimised if there is more than one sub-select. With the changes to the QueryJoinOptimizer it can now optimise both of these types of queries. The three comments above this one contain the query plan and the statistics. I managed to get my code to wait for the HDT merging to complete, but as you can see there are still quite a few places where the statistics are returning 0.0. I haven't looked further into why this is the case. Here is the final query that I ended up with: |
|
@D063520 Would you like to me comment on the original discussion page to outline my findings and the solution? |
|
yes please! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.