ERROR ON ELASTIC SEARCH 7.13.1

[dvdieu]

--------------- S U M M A R Y ------------

Command Line: -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-847774562637708686 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Des.cgroups.hierarchy.override=/ -Xms12g -Xmx12g -XX:MaxDirectMemorySize=6442450944 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true org.elasticsearch.bootstrap.Elasticsearch -Ebootstrap.memory_lock=true -Enode.name=esnode1 -Ecluster.initial_master_nodes=10.10.2.1, 10.10.2.2 -Enode.data=true -Ediscovery.seed_hosts=10.10.2.1, 10.10.2.2, 10.10.2.3 -Ecluster.name=es-docker-cluster -Enode.master=true

Host: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 8 cores, 23G, CentOS Linux release 8.4.2105
Time: Thu Aug 5 16:31:14 2021 UTC elapsed time: 65.623954 seconds (0d 0h 1m 5s)

--------------- T H R E A D ---------------

Current thread (0x00007f1e7c028e80): JavaThread "elasticsearch[esnode1][write][T#7]" daemon [_thread_in_native, id=286, stack(0x00007f1d17afb000,0x00007f1d17bfc000)]

Stack: [0x00007f1d17afb000,0x00007f1d17bfc000], sp=0x00007f1d17bf93b0, free space=1016k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x86101] cfree+0x21

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j com.coccoc.Tokenizer.initialize(Ljava/lang/String;)I+0
j com.coccoc.Tokenizer.(Ljava/lang/String;)V+6
j org.apache.lucene.analysis.vi.VietnameseTokenizerImpl.lambda$new$0(Lorg/elasticsearch/analysis/VietnameseConfig;)Lcom/coccoc/Tokenizer;+8
j org.apache.lucene.analysis.vi.VietnameseTokenizerImpl$$Lambda$5942+0x0000000801994660.run()Ljava/lang/Object;+4
J 2859 c2 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; java.base@16 (9 bytes) @ 0x00007f1f5bb8bfd0 [0x00007f1f5bb8bfa0+0x0000000000000030]
j org.apache.lucene.analysis.vi.VietnameseTokenizerImpl.(Lorg/elasticsearch/analysis/VietnameseConfig;Ljava/io/Reader;)V+73
j org.apache.lucene.analysis.vi.VietnameseTokenizer.(Lorg/elasticsearch/analysis/VietnameseConfig;)V+71
j org.elasticsearch.index.analysis.VietnameseTokenizerFactory.create()Lorg/apache/lucene/analysis/Tokenizer;+8
j org.elasticsearch.index.analysis.CustomAnalyzer.createComponents(Ljava/lang/String;)Lorg/apache/lucene/analysis/Analyzer$TokenStreamComponents;+4
j org.apache.lucene.analysis.AnalyzerWrapper.createComponents(Ljava/lang/String;)Lorg/apache/lucene/analysis/Analyzer$TokenStreamComponents;+8
j org.apache.lucene.analysis.AnalyzerWrapper.createComponents(Ljava/lang/String;)Lorg/apache/lucene/analysis/Analyzer$TokenStreamComponents;+8
j org.apache.lucene.analysis.Analyzer.tokenStream(Ljava/lang/String;Ljava/lang/String;)Lorg/apache/lucene/analysis/TokenStream;+58
J 8816 c1 org.apache.lucene.document.Field.tokenStream(Lorg/apache/lucene/analysis/Analyzer;Lorg/apache/lucene/analysis/TokenStream;)Lorg/apache/lucene/analysis/TokenStream; (188 bytes) @ 0x00007f1f5511d944 [0x00007f1f5511abc0+0x0000000000002d84]
j org.apache.lucene.index.DefaultIndexingChain$PerField.invert(ILorg/apache/lucene/index/IndexableField;Z)V+91
j org.apache.lucene.index.DefaultIndexingChain.processField(ILorg/apache/lucene/index/IndexableField;JI)I+113
J 10620 c2 org.apache.lucene.index.DefaultIndexingChain.processDocument(ILjava/lang/Iterable;)V (181 bytes) @ 0x00007f1f5c12b094 [0x00007f1f5c12ae20+0x0000000000000274]
J 10633 c2 org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(Ljava/lang/Iterable;Lorg/apache/lucene/index/DocumentsWriterDeleteQueue$Node;Lorg/apache/lucene/index/DocumentsWriter$FlushNotifications;)J (180 bytes) @ 0x00007f1f5c132aec [0x00007f1f5c132980+0x000000000000016c]
J 10436 c1 org.apache.lucene.index.DocumentsWriter.updateDocuments(Ljava/lang/Iterable;Lorg/apache/lucene/index/DocumentsWriterDeleteQueue$Node;)J (280 bytes) @ 0x00007f1f54d4573c [0x00007f1f54d455a0+0x000000000000019c]
j org.apache.lucene.index.IndexWriter.updateDocuments(Lorg/apache/lucene/index/DocumentsWriterDeleteQueue$Node;Ljava/lang/Iterable;)J+13
J 10062 c1 org.elasticsearch.index.engine.InternalEngine.addDocs(Ljava/util/List;Lorg/apache/lucene/index/IndexWriter;)V (49 bytes) @ 0x00007f1f5496e12c [0x00007f1f5496d960+0x00000000000007cc]
j org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(Lorg/elasticsearch/index/engine/Engine$Index;Lorg/elasticsearch/index/engine/InternalEngine$IndexingStrategy;)Lorg/elasticsearch/index/engine/Engine$IndexResult;+272
j org.elasticsearch.index.engine.InternalEngine.index(Lorg/elasticsearch/index/engine/Engine$Index;)Lorg/elasticsearch/index/engine/Engine$IndexResult;+418
J 10965 c2 org.elasticsearch.index.shard.IndexShard.index(Lorg/elasticsearch/index/engine/Engine;Lorg/elasticsearch/index/engine/Engine$Index;)Lorg/elasticsearch/index/engine/Engine$IndexResult; (316 bytes) @ 0x00007f1f5c1e258c [0x00007f1f5c1e22e0+0x00000000000002ac]
j org.elasticsearch.index.shard.IndexShard.applyIndexOperation(Lorg/elasticsearch/index/engine/Engine;JJJLorg/elasticsearch/index/VersionType;JJJZLorg/elasticsearch/index/engine/Engine$Operation$Origin;Lorg/elasticsearch/index/mapper/SourceToParse;)Lorg/elasticsearch/index/engine/Engine$IndexResult;+230
j org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(JLorg/elasticsearch/index/VersionType;Lorg/elasticsearch/index/mapper/SourceToParse;JJJZ)Lorg/elasticsearch/index/engine/Engine$IndexResult;+49
j org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(Lorg/elasticsearch/action/bulk/BulkPrimaryExecutionContext;Lorg/elasticsearch/action/update/UpdateHelper;Ljava/util/function/LongSupplier;Lorg/elasticsearch/action/bulk/MappingUpdatePerformer;Ljava/util/function/Consumer;Lorg/elasticsearch/action/ActionListener;)Z+456
j org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun()V+45
J 11088 c1 org.elasticsearch.common.util.concurrent.AbstractRunnable.run()V (32 bytes) @ 0x00007f1f5530740c [0x00007f1f55307300+0x000000000000010c]
j org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(Lorg/elasticsearch/action/bulk/BulkShardRequest;Lorg/elasticsearch/index/shard/IndexShard;Lorg/elasticsearch/action/update/UpdateHelper;Ljava/util/function/LongSupplier;Lorg/elasticsearch/action/bulk/MappingUpdatePerformer;Ljava/util/function/Consumer;Lorg/elasticsearch/action/ActionListener;Lorg/elasticsearch/threadpool/ThreadPool;Ljava/lang/String;)V+21
j org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(Lorg/elasticsearch/action/bulk/BulkShardRequest;Lorg/elasticsearch/index/shard/IndexShard;Lorg/elasticsearch/action/ActionListener;)V+71
j org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(Lorg/elasticsearch/action/support/replication/ReplicatedWriteRequest;Lorg/elasticsearch/index/shard/IndexShard;Lorg/elasticsearch/action/ActionListener;)V+7
j org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun()V+16
j org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun()V+24
J 11088 c1 org.elasticsearch.common.util.concurrent.AbstractRunnable.run()V (32 bytes) @ 0x00007f1f5530740c [0x00007f1f55307300+0x000000000000010c]
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@16
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@16
j java.lang.Thread.run()V+11 java.base@16
v ~StubRoutines::call_stub

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0xfffffffffffffff7

Register to memory mapping:

RAX=0x0 is NULL
RBX=0x00007f1e4c00a620 points into unknown readable memory: 0xffffffffffffffff | ff ff ff ff ff ff ff ff
RCX=0x00007f1f71c90999: <offset 0x0000000000080999> in /lib64/libc.so.6 at 0x00007f1f71c10000
RDX=0x00007f1e4c000080 points into unknown readable memory: 0x00007f1e4c034220 | 20 42 03 4c 1e 7f 00 00
RSP=0x00007f1d17bf93b0 is pointing into the stack for thread: 0x00007f1e7c028e80
RBP=0x00007f1e9413a040 points into unknown readable memory: 0x00007f1e40009110 | 10 91 00 40 1e 7f 00 00
RSI=0x0 is NULL
RDI=0xffffffffffffffff is an unknown value
R8 =0x00007f1e5408319e points into unknown readable memory: 07 00
R9 =0x0000000000000007 is an unknown value
R10=0x0 is NULL
R11=0x0000000000000202 is an unknown value
R12=0x0 is NULL
R13=0x00007f1e54083ea0 points into unknown readable memory: 0x00000000fbad2488 | 88 24 ad fb 00 00 00 00
R14=0x0000000000000018 is an unknown value
R15=0x00007f1d17bf9460 is pointing into the stack for thread: 0x00007f1e7c028e80

[bachan]

Thanks for report. It would help if you provide a minimal example to reproduce it. If it's hard to come up with such minimal example, please share at least part of your index and a set of requests that you are using.