Positions are enabled on analyzed string fields by default, so that phrase queries will work out of the box. The more often that a term appears, the more space that is needed to store its position data. Very common words, by definition, appear very commonly and their positions data can run to megabytes or gigabytes on large corpuses.

Running a phrase query on a high frequency word like the might result in gigabytes of data being read from disk. That data will be stored in the kernel file system cache to speed up later access, which seems like a good thing, but it might cause other data to be evicted from the cache which will slow down subsequent queries.

This is clearly a problem that needs solving.

The first question you should ask yourself is: ``Do you need phrase or proximity queries?''

Often, the answer is no. For many use cases, such as logging, you need to know whether a term appears in a document — information which is provided by the postings list — but not where it appears. Or perhaps you need to use phrase queries on one or two fields, but you can disable positions data on all of the other analyzed string fields.

The index_options parameter allows you to control what information is stored in the index for each field. Valid values are:

docs

Only store which documents contain which terms. This is the default for not_analyzed string fields.

freqs

Store docs information, plus how often each term appears in each document. Term frequencies are needed for a complete TF/IDF relevance calculations, but they are not required if you just need to know whether a document contains a particular term or not.

positions

Store docs and freqs, plus the position of each term in each document. This is the default for analyzed string fields, but can be disabled if phrase/proximity matching is not needed.

offsets

Store docs, freqs, positions and the start and end character offsets of each term in the original string. This information is used by the {ref}search-request-highlighting.html#postings-highlighter[postings highlighter] but is disabled by default.

You can set index_options on fields added at index creation time, or when adding new fields using the put-mapping API. This setting can’t be changed on existing fields:

1. The title field uses the default setting of positions, so it is suitable for phrase/proximity queries.

2. The content field has positions disabled and so cannot be used for phrase/proximity queries.

Removing stopwords is one way of reducing the size of the positions data quite dramatically. An index with stopwords removed can still be used for phrase queries because the original positions of the remaining terms is maintained, as we saw in [maintaining-positions]. But of course, excluding terms from the index reduces searchability. We wouldn’t be able to differentiate between the two phrases Man in the moon'' and Man on the moon''.

Fortunately, there is a way to have our cake and eat it: the common_grams token filter.