elastic 불용어 테스트
DELETE my_index_3
PUT my_index_3
{
"settings": {
"analysis": {
"analyzer": {
"my_stop_analyzer": {
"type": "stop",
"stopwords_path" : "analysis/stopword.txt"
}
}
}
}
}
POST /my_index_3/_refresh
POST my_index_3/_analyze
{
"analyzer": "my_stop_analyzer",
"text": "가다"
}
'ELK > elasticsearch' 카테고리의 다른 글
elasticsearch : count (0) | 2019.05.21 |
---|---|
elasticsearch + shell01 (0) | 2019.05.11 |
python + bulk + insert + code (0) | 2019.04.24 |
elastic bacis#3 (0) | 2019.04.23 |
elastic basic #2 (0) | 2019.04.22 |
python + bulk + insert + code
from elasticsearch import helpers, Elasticsearch
import csv
es = Elasticsearch(hosts="192.168.2.10")
with open(r"C:\Users\user\Desktop\comm_kkc.csv") as f:
reader = csv.DictReader(f)
helpers.bulk(es, reader, index="tb_frip_srch_tour", doc_type="doc")
f.close()
'ELK > elasticsearch' 카테고리의 다른 글
elasticsearch + shell01 (0) | 2019.05.11 |
---|---|
elastic 불용어 테스트 (0) | 2019.05.01 |
elastic bacis#3 (0) | 2019.04.23 |
elastic basic #2 (0) | 2019.04.22 |
elastic basic #1 (0) | 2019.04.22 |
elastic bacis#3
#1 . index create
#!/bin/bash
curl -X PUT 'http://192.168.240.10:9200/test_index_1?pretty' -H 'Content-Type: application/json' -d '
{
"settings" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
},
"mappings" : {
"doc" : {
"properties" : {
"age" : { "type" : "integer" },
"name" : { "type" : "text" }
}
}
}
}'
#2 .data Insert
#!/bin/bash
curl -X POST 'http://192.168.240.10:9200/test_index_1/doc?pretty' -H 'Content-Type: application/json' -d '
{
"age" : 31,
"name" : "kimjh"
}'
#3 update
- field 추가
onfSE2oBZ6r8F2QOkMRS
#!/bin/bash
curl -X POST 'http://192.168.240.10:9200/test_index_1/doc/onfSE2oBZ6r8F2QOkMRS/_update?pretty' -H 'Content-Type: application/json' -d '
{
"doc" : {
"phone" : "032-000-0000"
}
}'
---------------------------------
#!/bin/bash
curl -X POST 'http://192.168.240.10:9200/test_index_1/doc/onfSE2oBZ6r8F2QOkMRS/_update?pretty' -H 'Content-Type: application/json' -d '
{
"script" : "ctx._source.age += 5"
}'
'ELK > elasticsearch' 카테고리의 다른 글
elastic 불용어 테스트 (0) | 2019.05.01 |
---|---|
python + bulk + insert + code (0) | 2019.04.24 |
elastic basic #2 (0) | 2019.04.22 |
elastic basic #1 (0) | 2019.04.22 |
python ( elasticsearch - search template ) (0) | 2019.04.20 |
elastic basic #2
# 설명은 주말에
GET /navr-rank/_search
{
"size" : 20,
"sort": [
{
"order": {
"order": "asc"
}
}
],
"query": {
"match": {
"date": "2019-04-20"
}
}
}
'ELK > elasticsearch' 카테고리의 다른 글
python + bulk + insert + code (0) | 2019.04.24 |
---|---|
elastic bacis#3 (0) | 2019.04.23 |
elastic basic #1 (0) | 2019.04.22 |
python ( elasticsearch - search template ) (0) | 2019.04.20 |
Elastic basic#01 (0) | 2019.04.19 |
elastic basic #1
GET /navr-rank/_search
{
"sort": [
{
"order": {
"order": "desc"
}
}
],
"query" : {
"range" : {
"date" : {
"gte": "now-2d/d",
"lt" : "now-1d/d"
}
}
}
}
'ELK > elasticsearch' 카테고리의 다른 글
elastic bacis#3 (0) | 2019.04.23 |
---|---|
elastic basic #2 (0) | 2019.04.22 |
python ( elasticsearch - search template ) (0) | 2019.04.20 |
Elastic basic#01 (0) | 2019.04.19 |
특정 데이터 count (0) | 2019.04.10 |
python ( elasticsearch - search template )
def searchTemplate(self, searchWord):
response = \
self.elanode.search_template (
index="navr-rank",
body= {
"inline" : {
"query" : {
"match" : {"{{field}}" : "{{value}}"}
}
},
"params" : {
"field" : "text", "value" : searchWord
}
}
)
'ELK > elasticsearch' 카테고리의 다른 글
elastic basic #2 (0) | 2019.04.22 |
---|---|
elastic basic #1 (0) | 2019.04.22 |
Elastic basic#01 (0) | 2019.04.19 |
특정 데이터 count (0) | 2019.04.10 |
Elasticsearch basic01 (0) | 2019.04.02 |
Elastic basic#01
데이터 구조
- 인덱스 (index)
- 타입 (type)
- 도큐먼트 (document)
- 인덱스 : 샤드 (shard) + 복사본 (replica)
관계형 db 엘라스틱서치
---------------------------------------
데이터 베이스 인덱스
테이블 타입 (type)
열(Colume) 도큐먼트(document)
행(Row) 필드(field)
스키마(Schema) 매핑(Mapping)
curl -X {method} http://host:9200/{index}/{type}/{id} -H 'Content-Type: application/json' -d '{ ...}'
---------------------------------------
HTTP Method CRUD SQL
GET Read Select
PUT Update Update
POST Create Insert
DELETE Delete Delete
'ELK > elasticsearch' 카테고리의 다른 글
elastic basic #1 (0) | 2019.04.22 |
---|---|
python ( elasticsearch - search template ) (0) | 2019.04.20 |
특정 데이터 count (0) | 2019.04.10 |
Elasticsearch basic01 (0) | 2019.04.02 |
elastisearch java api (0) | 2019.03.26 |
특정 데이터 count
GET test-index/doc/_count
{
"query" : {
"term" : { "data" : "고양이" }
}
}
========================
python code
def documentCount(self):
result = self._client.count (
index= "stu-index-1",
doc_type= "doc",
body= {
"query" : {
"match" : {"y" : "11"}
}
}
)
print (result)
'ELK > elasticsearch' 카테고리의 다른 글
python ( elasticsearch - search template ) (0) | 2019.04.20 |
---|---|
Elastic basic#01 (0) | 2019.04.19 |
Elasticsearch basic01 (0) | 2019.04.02 |
elastisearch java api (0) | 2019.03.26 |
elasticsearch python 주말에 정리할 것 (0) | 2019.03.18 |
Elasticsearch basic01
Elastic search port : 9200 를 사용 한다. 참고로 (Kibana는 5601)
index-name : movie-index
---------------------------------
term의 사전적 의미 "용어, 말"
아래 코드를 해석해보자면 movie-index의 필드 movie_jangre 에 "드라마"라는 용어가 들어있는 document를 찾아줘
curl -X GET "localhost:9200/movie-index/_search?pretty" -H 'Content-Type: application/json' -d '
{
"query" : {
"term" : {
"movie_jangre" : "드라마"
}
}
}'
자바 api
MatchQueryBuilder matchQueryBuilder = matchQuery("movie_jangre", searchWord);
FROM/SIZE
"시작 offset 0에서 부터 10개를 가져와서 조회해라"
curl -X GET '192.168.240.141:9200/movie-index/_search?pretty' -H 'Content-Type: application/json' -d '
{
"from" : 0,
"size" : 10,
"query" : {
"match" : {
"movie_jangre" : "드라마"
}
}
}'
java api
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.query(matchQueryBuilder)
.from(0)
.size(20)
.sort(new FieldSortBuilder("movie_rank").order(SortOrder.ASC));
'ELK > elasticsearch' 카테고리의 다른 글
Elastic basic#01 (0) | 2019.04.19 |
---|---|
특정 데이터 count (0) | 2019.04.10 |
elastisearch java api (0) | 2019.03.26 |
elasticsearch python 주말에 정리할 것 (0) | 2019.03.18 |
공부중 (0) | 2019.03.18 |
elastisearch java api
package com.app.elasticsrv;
import org.apache.lucene.queryparser.flexible.core.builders.QueryBuilder;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.index.search.MatchQuery;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.aggregations.AggregationBuilder;
import org.elasticsearch.search.aggregations.AggregationBuilders;
import org.elasticsearch.search.aggregations.BucketOrder;
import org.elasticsearch.search.aggregations.bucket.range.AbstractRangeBuilder;
import org.elasticsearch.search.aggregations.bucket.range.RangeAggregationBuilder;
import org.elasticsearch.search.aggregations.bucket.range.RangeAggregator.Range;
import org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityAggregationBuilder;
import org.elasticsearch.search.aggregations.metrics.max.MaxAggregationBuilder;
import org.elasticsearch.search.aggregations.metrics.min.MinAggregationBuilder;
import org.elasticsearch.search.aggregations.metrics.sum.SumAggregationBuilder;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import static org.elasticsearch.index.query.QueryBuilders.*;
import java.io.FileFilter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import com.app.util.Utils;
import com.cedarsoftware.util.io.JsonWriter;
import com.fasterxml.jackson.core.json.JsonWriteContext;
public class Srcher {
private String keyWord = null; // 검색 대상
private String elastIndex = null; // 검색 대상 index
private RestHighLevelClient _client = null;
private SearchRequest searchRequest = null; // 요청
private SearchResponse searchResponse = null; // 응답
private int from; // pagination
private int size; // pagination
private Srcher(Builder builder) {
this.keyWord = builder.keyWord;
this.elastIndex = builder.elastIndex;
this.from = builder.from;
this.size = builder.size;
this.searchRequest = builder.searchRequest;
this._client = new Elasrver().srvSetting();
}
public void searchDefault() {
System.out.println("tst : " + this.keyWord);
this.searchRequest = new SearchRequest(this.elastIndex);
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.query(matchQuery("name", this.keyWord))
.from(this.from)
.size(this.size)
.timeout(new TimeValue(60, TimeUnit.SECONDS));
// request
searchRequest.source(sourceBuilder);
// response
try {
this.searchResponse = this._client.search(this.searchRequest, RequestOptions.DEFAULT);
SearchHits hits = this.searchResponse.getHits();
for (SearchHit hit : hits) {
System.out.println(hit.getSourceAsString());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} // end of searchDefault function
/**
* aggregation : Cardinality
*/
public void resultAggregation() {
// 날짜를 기준으로 aggregation을 한다면 추천 단어로 사용할 수 있다.
CardinalityAggregationBuilder aggregation = AggregationBuilders
.cardinality("age_count")
.field("age");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.aggregation(aggregation);
// request
searchRequest.source(sourceBuilder);
// response
try {
this.searchResponse = this._client.search(this.searchRequest, RequestOptions.DEFAULT);
SearchHits hits = this.searchResponse.getHits();
for (SearchHit hit : hits) {
System.out.println(hit.getSourceAsString());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} // end of resultAggregation function
/**
* sum aggregation
*/
public void sumAggregationTest() {
AggregationBuilder aggregation = AggregationBuilders
.sum("total_age")
.field("age");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.query(matchQuery("name", "kim"))
.aggregation(aggregation);
// 요청
searchRequest.source(sourceBuilder);
try {
this.searchResponse = this._client.search(this.searchRequest, RequestOptions.DEFAULT);
// JSON 으로 변환
JSONObject jsonObj = Utils.fromStringToJson(this.searchResponse.toString());
System.out.println(jsonObj.get("aggregations"));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} // end of resultAggregation function
public void testAggregation() {
AggregationBuilder aggregation = AggregationBuilders
.cardinality("age_count")
.field("age")
.precisionThreshold(100);
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.aggregation(aggregation);
// 요청
searchRequest.source(sourceBuilder);
try {
this.searchResponse = this._client.search(this.searchRequest, RequestOptions.DEFAULT);
// JSON 으로 변환
JSONObject jsonObj = Utils.fromStringToJson(this.searchResponse.toString());
System.out.println(jsonObj.get("aggregations"));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
// ===============================================================================================
// 추천 키워드
// =================================
public void dateAggregationTest() {
/**
* cardinality로 date 별로 aggregation을 진행한다.
*/
AggregationBuilder aggregation = AggregationBuilders
.cardinality("date_count")
.field("date")
.precisionThreshold(100);
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.aggregation(aggregation);
Map<String, Object> responseData = null;
Map<String, String> tmpData = null;
Set<String> dateCollectSet = new HashSet<String>(); // 날짜 데이터를 담는다.
// 요청
searchRequest.source(sourceBuilder);
try {
this.searchResponse = this._client.search(this.searchRequest, RequestOptions.DEFAULT);
// JSON 으로 변환 -----------------------------------------------------------------
JSONObject jsonObj = Utils.fromStringToJson(this.searchResponse.toString());
JSONObject resltObj = (JSONObject) jsonObj.get("hits");
JSONArray hitObj = (JSONArray) resltObj.get("hits");
// -----------------------------------------------------------------------------
for (int i = 0; i < hitObj.size(); i++) {
responseData = (Map<String, Object>) hitObj.get(i);
tmpData = (Map<String, String>) responseData.get("_source");
//System.out.println(tmpData.get("date"));
dateCollectSet.add(tmpData.get("date"));
}
dateCollectSet.forEach(dateParam -> {
JSONObject resultObj = null;
AggregationBuilder sumAggregation = null;
SearchSourceBuilder sourceBuilderSum = null;
Map<String, Object> dataDict = null;
long countValue;
// =================================================================
sumAggregation = AggregationBuilders // <=== sum aggregation
.sum("total_date")
.field("date");
// =================================================================
sourceBuilderSum = new SearchSourceBuilder()
.query(matchQuery("date", dateParam))
.aggregation(sumAggregation);
// 요청
searchRequest.source(sourceBuilderSum);
try {
this.searchResponse = this._client.search(this.searchRequest, RequestOptions.DEFAULT);
// JSON 으로 변환
resultObj = Utils.fromStringToJson(this.searchResponse.toString());
dataDict = (Map<String, Object>) resultObj.get("hits");
countValue = (long) dataDict.get("total");
System.out.println(countValue + " : " + dateParam);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
});
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
/*
AggregationBuilder aggregation = AggregationBuilders
.range("date aggregation")
.field("date")
.format("yyyy-MM-dd")
.addRange("2019-03-25", );
*/
} // end of dateAggregationTest function
/**
* ==============================================================================
* date count
* 2019-03-26
* ==============================================================================
*/
public void recommandQueryTest1() {
// tophit
BucketOrder order = null;
AggregationBuilder aggregation = AggregationBuilders
.terms("date_info")
.field("date")
.order(order.count(false));
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.aggregation(aggregation);
searchRequest.source(sourceBuilder);
try {
this.searchResponse = this._client.search(this.searchRequest, RequestOptions.DEFAULT);
// JSON 으로 변환
JSONObject jsonObj = Utils.fromStringToJson(this.searchResponse.toString());
System.out.println(jsonObj);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} // end of resultAggregation function
/**
* ==============================================================================
* Recommand demo code
* Aggregation Query (date range)
* 2019-03-26
* @author KimJunHyeon
* @param
* ==============================================================================
*/
public void recommandQueryTest2() {
AggregationBuilder aggregation = AggregationBuilders
.dateRange("date_range")
.field("date")
.format("yyyy-MM-dd")
.addUnboundedTo("now+2d/d")
.addUnboundedFrom("now-10d/d")
.subAggregation(
AggregationBuilders
.terms("words")
.field("word")
.size(2)
.order(BucketOrder.count(false))
);
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.aggregation(aggregation);
// Elastic search server request
searchRequest.source(sourceBuilder);
try {
this.searchResponse = this._client.search(this.searchRequest, RequestOptions.DEFAULT);
JSONObject jsonObj = Utils.fromStringToJson(this.searchResponse.toString());
JSONObject dateRageObj = (JSONObject) jsonObj.get("aggregations");
Map<String, Map<String, Object>> testElement = dateRageObj;
JSONArray jsonArry = (JSONArray) testElement.get("date_range#date_range").get("buckets");
JSONObject tmpObj = null;
Map<String, Object> result = null;
for (int i = 0; i < jsonArry.size(); i++) {
tmpObj = (JSONObject) jsonArry.get(i);
result = (Map<String, Object>) tmpObj.get("sterms#words");
JSONArray jsonArrySub = (JSONArray) result.get("buckets");
for (int j = 0; j < jsonArrySub.size(); j++) {
JSONObject jsonObject = (JSONObject) jsonArrySub.get(j);
System.out.println(jsonObject.get("key"));
}
System.out.println(jsonArrySub);
//System.out.println(JsonWriter.formatJson(tmpObj.get("sterms#words").toString()));
}
//System.out.println(JsonWriter.formatJson(jsonObj.get("aggregations").toString()));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} // end of resultAggregation function
/**
* aggregation : 최댓값
*/
public void rangeAggregationTest2() {
AggregationBuilder aggregation = AggregationBuilders
.range("age_count")
.field("age")
.addRange(10.0, 12.0);
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
.query(matchQuery("name", "kim"))
.aggregation(aggregation);
// 요청
searchRequest.source(sourceBuilder);
try {
this.searchResponse = this._client.search(this.searchRequest, RequestOptions.DEFAULT);
// JSON 으로 변환
JSONObject jsonObj = Utils.fromStringToJson(this.searchResponse.toString());
System.out.println(jsonObj.get("aggregations"));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} // end of resultAggregation function
public void clientNodeDie() {
if (_client != null) {
try {
_client.close();
System.out.println("client node die !!");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
/**
*
* @author user
* builder pattern
*/
public static class Builder {
private String keyWord = null; // 검색 대상
private String elastIndex = null; // 검색 대상 index
private SearchRequest searchRequest = null;
private int from; // pagination
private int size; // pagination
/**
* 검색 대상
* @param keyWord
*/
public Builder(String keyWord) {
if (keyWord == null) {
throw new IllegalArgumentException("keyWord can not be null");
}
else {
this.keyWord = keyWord;
}
}
/**
* 검색 페이지 설정
* @param from
* @param size
* @return
*/
public Builder withPagination(int from, int size) {
this.from = from;
this.size = size;
return this;
}
/**
* 검색 인덱스 설정
* @param elastIndex
* @return
*/
public Builder withIndex (String elastIndex) {
this.elastIndex = elastIndex;
searchRequest = new SearchRequest(this.elastIndex);
return this;
}
/**
*
* @return
*/
public Srcher build() {
return new Srcher(this);
}
}
}
'ELK > elasticsearch' 카테고리의 다른 글
특정 데이터 count (0) | 2019.04.10 |
---|---|
Elasticsearch basic01 (0) | 2019.04.02 |
elasticsearch python 주말에 정리할 것 (0) | 2019.03.18 |
공부중 (0) | 2019.03.18 |
java api - nestedQuery (0) | 2019.03.12 |