Without an fsync to flush data in the file-system cache to disk, we cannot be sure that the data will still be there after a power failure, or even after exiting the application normally. For Elasticsearch to be reliable, it needs to ensure that changes are persisted to disk.

In [dynamic-indices] we said that a full commit flushes segments to disk and writes a commit point, which lists all known segments. Elasticsearch uses this commit point during startup or when reopening an index to decide which segments belong to the current shard.

While we refresh once every second to achieve near real-time search, we still need to do full commits regularly to make sure that we can recover from failure. But what about the document changes that happen between commits? We don’t want to lose those either.

Elasticsearch added a translog or transaction log, which records every operation in Elasticsearch as it happens. With the translog, the process now looks like this:

1. When a document is indexed, it is added to the in-memory buffer AND appended to the translog.

   

   Figure 1. New documents are added to the in-memory buffer and appended to the transaction log

2. Once every second, the shard is refreshed:

   • The docs in the in-memory buffer are written to a new segment, without an fsync.

   • The segment is opened to make it visible to search.

   • The in-memory buffer is cleared.

   

   Figure 2. After a refresh, the buffer is cleared but the transaction log is not

3. This process continues with more documents being added to the in-memory buffer and appended to the transaction log.

   

   Figure 3. The transaction log keeps accumulating documents

4. Every so often — such as when the translog is getting too big — the index is flushed: a new translog is created and a full commit is performed:

   • Any docs in the in-memory buffer are written to a new segment.

   • The buffer is cleared.

   • A commit point is written to disk.

   • The file-system cache is flushed with an fsync.

   • The old translog is deleted.

   

   Figure 4. After a flush, the segments are fully commited and the transaction log is cleared

The translog provides a persistent record of all operations that have not yet been flushed to disk. When starting up, Elasticsearch will use the last commit point to recover known segments from disk, and will then replay all operations in the translog to add the changes that happened after the last commit.

The translog is also used to provide real-time CRUD. When you try to retrieve, update, or delete a document by ID, it first checks the translog for any recent changes before trying to retrieve the document from the relevant segment. This means that it always has access to the latest known version of the document, in real-time.

flush API

POST /blogs/_flush (1)

POST /_flush?wait_for_ongoing (2)