Could search accept hyphens?

[normandy7]

It seems that although the hyphen (-) is not configured as a search separator, it acts this way in the query. In our documentation, users would generally be more likely to deliberately search for (or expect results with) hyphen constructions, such as names of libraries.

Any idea what's going on there? Is the issue on the Lunr side? Would there be a way to circumvent it?

[squidfunk]

How did you configure separator? Note that by default, hyphens are included. If you changed to a separator that doesn't include a hyphen, and you still can't use them in your documentation, please submit a bug report with a minimal reproduction.

[normandy7]

Right now at https://docs.neptune.ai/ we're using the following separator:

[\s\-,:!=\[\]()"/]+|\.(?!\d)

When you search for on-prem, it seems to only take the "prem" part into account. When I experimented with the otherwise same separator but with \- left out, it does successfully match hyphenated compounds, but it doesn't change the fact that the hyphen seems to break the query itself somehow.

It's especially interesting to follow the results as you type out on-prem one character at a time. I did some troubleshooting with each separator version in parallel.

1. When you type just o in the search box (works as expected):
   1. Hyphen included as separator: Search box suggestion becomes "october". Only words beginning with "o" are matched.
   2. Hyphen excluded: Search box suggestion becomes "october". Words beginning with "o" are matched, including hyphen compounds as a whole.
2. on
   1. Hyphen included: Search suggestion becomes "once".
   2. Hyphen excluded: Search suggestion becomes "on-premises". The top result is different (and related to on-prem). The problem is that following the suggestion yields unexpected results.
3. on- -- something quite strange happens here. In each scenario, the match is very broad and has nothing to do with "on".
4. on-p -- now it's as if the "on" part doesn't exist. The results are the same as for just p.
5. Typing out all the way to on-prem seems to confirm that only the part after the hyphen is considered.
   1. Hyphen included: We're still fine thanks to our separator, as the words "prem" or "premises" don't really occur in any other contexts.
   2. Hyphen excluded: There's no instance of "prem" or "premises" that doesn't have "on-" preceding it, so now search comes up empty.

Sorry for the long explanation, but wanted to clarify that the separator itself seems to work fine; it's just that querying doesn't work as expected.

[squidfunk]

Perfect, could you please create a bug report with a reproduction as requested in #5247 (comment)? I'll look into it then.

[squidfunk]

As an idea where this is coming from:

Regarding 3.: It might be that this is tokenized into "on" and "", the latter of which might explain the broad match
Regarding 4.: "on" is likely filtered because it's a stopword.

Regardless, a reproduction allows us to troubleshoot this.

[normandy7]

I was going to return to this now, but discovered that you have a whole new search in the works. Perhaps it's worth seeing if the issue persists before investigating this further? :)

[squidfunk]

Jup! I expect the new search to be much, much better and more flexible than the current one we have ☺️

[Stephen-Gates]

Did this get resolved? I get similar results to @normandy7

I use grave locations, Portion-Section-Grave format, on many of my pages e.g. 1-23-456.

Where:.

• Portion could be 1 or 2 digits, sometimes followed by the letter A.
• Section is 1 to 3 digits.
• Grave is either:
  • 1 to 3 digits
  • 1 to 3 digits / 1 to 3 digits - representing multiple graves together as one, e.g. 12/13
  • 1 to 3 digits, sometimes followed by a letter

The cemetery numbering scheme got whacky as they tried to squeeze more people in! So 1A-234-456A/457A is valid.

I'd like to search for, and the type-ahead to find, relevant pages when I type a grave location, e.g.

• typing 12- in search, finds people in Portion 12.
• typing 12-3- in search, finds people in Portion 12 Section 3.
• typing 12-3-4A in search, finds the person buried in Portion 12 Section 3 Grave 4A.

I'm no Regex expert but I thought removing the \- from the special characters would work. Now I'm just guessing on how I might achieve this.

Suggestions welcome.

[kamilkrzyskow]

iirc it's not resolved with the current lunr.js implementation, related discussion where some investigation was made:

• Search: whole phrase like "a-b.log" #6885

[squidfunk]

Correct, it's not yet resolved because of #6885 (reply in thread) (2.), i.e., the query tokenizer being different from the tokenizer used when indexing. We're currently in feature freeze due to the large update we're working on, so I don't think it can be solved at this moment in time. I'm sorry for the inconvenience and the wait, we're working on it.