Handling conflicts of elastic search

By Rajdeep Kaur

Recently I’ve set up an ELK stack and started feeding it data from application logs. Unfortunately one (and the most important) field was assigned a wrong mapping in Elasticsearch and instead of double it was sometimes interpreted as a string field.

I didn’t have much success googling a quick and easy solution, so here is one.

Kibana was reporting the following error:

Mapping conflict! A field is defined as several types (string, integer, etc) across the indices that match this pattern. You may still be able to use these conflict fields in parts of Kibana, but they will be unavailable for functions that require Kibana to know their type. Correcting this issue will require reindexing your data.

Problem arises with the following situation

As the message says, the problem is that a field has different type in different indices in Elasticsearch. By default Elasticsearch creates a new index every day. So every day the values are parsed by Elasticsearch and mapped to data types.

so I followed the following solution to resolve the conflict. my assumed conflicted index in this case is “index-2018.06.09”

  1. Downloading the index file of that day with the following command

    2. GET http://localhost:9200/index-2018.06.09?pretty

    The above command will give us the mapping of the particular index in the json formate.

    • Remove the second line which has index written in it.
    • remove the data inside the key settings(at the end of the file).
    • typecast the key to it’s required type(In my case I deleted the event and removed the key)
  2. Making an new index with the following command.

    3. PUT http://localhost:9200/index-2018.06.09-backup with the updated mapping as an attachment from previous step

  3. Reindex the old index with the new index

    4. POST http://localhost:9200/_reindexwith the following request body { “source”: { “index” : “index-2018.06.09” },”dest” : { “index” : “index-2018.06.09-back” } }

  4. Delete the previous index.

    5. DELETE http://localhost:9200/index-2018.06.09

    in order to make the consistency in the indices, we can make alias of new index to point old or again reindex so new index points old index.

    At last refresh the refresh the fields in kibana.

    Cheers !!!!!!!.