Previous kafka versions used sparse files for indexes and timeindexes. When using sparse files disk space is consumed only for the non-empty data blocks. This saves disk space, but when the disk becomes full this results in a failure when trying to write to a previously empty region of file. This in turn caused fatal integrity errors for Kafka.

In order to avoid this problem newer kafka versions allocate entire index files when new topic partition is created. This guarantees that when needed the indexes can always be written to.

Did this answer your question?