Usually, documents that mix multiple languages in a single field come from sources beyond your control, such as pages scraped from the Web:
{ "body": "Page not found / Seite nicht gefunden / Page non trouvée" }
They are the most difficult type of multilingual document to handle correctly.
Although you can simply use the standard
analyzer on all fields, your documents
will be less searchable than if you had used an appropriate stemmer. But of
course, you can’t choose just one stemmer—stemmers are language specific.
Or rather, stemmers are language and script specific. As discussed in
[different-scripts], if every language uses a different script, then
stemmers can be combined.
Assuming that your mix of languages uses the same script such as Latin, you have three choices available to you:
Split into separate fields
Analyze multiple times
Use n-grams
The Compact Language Detector mentioned in [identifying-language] can tell you which parts of the document are in which language. You can split up the text based on language and use the same approach as was used in [one-lang-fields].
If you primarily deal with a limited number of languages, you could use multi-fields to analyze the text once per language:
PUT /movies
{
"mappings": {
"title": {
"properties": {
"title": { (1)
"type": "string",
"fields": {
"de": { (2)
"type": "string",
"analyzer": "german"
},
"en": { (2)
"type": "string",
"analyzer": "english"
},
"fr": { (2)
"type": "string",
"analyzer": "french"
},
"es": { (2)
"type": "string",
"analyzer": "spanish"
}
}
}
}
}
}
}
The main title
field uses the standard
analyzer.
Each subfield applies a different language analyzer
to the text in the title
field.
You could index all words as n-grams, using the same approach as described in [ngrams-compound-words]. Most inflections involve adding a suffix (or in some languages, a prefix) to a word, so by breaking each word into n-grams, you have a good chance of matching words that are similar but not exactly the same. This can be combined with the analyze-multiple times approach to provide a catchall field for unsupported languages:
PUT /movies
{
"settings": {
"analysis": {...} (1)
},
"mappings": {
"title": {
"properties": {
"title": {
"type": "string",
"fields": {
"de": {
"type": "string",
"analyzer": "german"
},
"en": {
"type": "string",
"analyzer": "english"
},
"fr": {
"type": "string",
"analyzer": "french"
},
"es": {
"type": "string",
"analyzer": "spanish"
},
"general": { (2)
"type": "string",
"analyzer": "trigrams"
}
}
}
}
}
}
}
In the analysis
section, we define the same trigrams
analyzer as described in [ngrams-compound-words].
The title.general
field uses the trigrams
analyzer
to index any language.
When querying the catchall general
field, you can use
minimum_should_match
to reduce the number of low-quality matches. It may
also be necessary to boost the other fields slightly more than the general
field, so that matches on the main language fields are given more weight
than those on the general
field:
GET /movies/movie/_search
{
"query": {
"multi_match": {
"query": "club de la lucha",
"fields": [ "title*^1.5", "title.general" ], (1)
"type": "most_fields",
"minimum_should_match": "75%" (2)
}
}
}
All title
or title.*
fields are given a slight boost over the
title.general
field.
The minimum_should_match
parameter reduces the number of low-quality matches returned, especially important for the title.general
field.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。