es可以用来存储日志,一般日志存储只是短期保存,超过一定时间日志要是能自动删除最好,这样保证索引文档不会过多,查询时效性也能得到保证。
索引的生命周期分为四个阶段:HOT->WARM->COLD->DELETE。
上面除了HOT为必须的阶段外,其他为非必须阶段,可以任意选择配置。因为日志索引只要满足自己删除功能,所以下文只配置了HOT与DELETE阶段。
三步实现完成es生命周期管理
配置策略(policy)、索引模版(template)、索引(index)
1,配置策略(policy)
PUT _ilm/policy/datastream_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover":{
"max_docs":1
}
}
},
"delete": {
"min_age": "30s",
"actions": {
"delete": {}
}
}
}
}
}10.11.12.13.14.15.16.17.18.20.
创建了一个 “datastream_policy”策略,策略含义是:当hot阶段索引文档超过一条,就会发生rollover——便于测试,
delete阶段,超过30s执行删除动作。
还可以选择其它维度控制,索引大小,索引时间:
2,索引模版(template)
PUT _template/datastream_template
{
"index_patterns": ["datastream-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "datastream_policy",
"index.lifecycle.rollover_alias": "datastream"
}
}10.
创建了一个索引模板,"index_patterns": ["datastream-*"],索引以datastream-开头的则自动采用setting的配置;
"index.lifecycle.name": "datastream_policy", 采用”datastream_policy“的策略;
"index.lifecycle.rollover_alias": "datastream",创建使用该模版创建的索引统一用”datastream“的别名进行管理
3,索引(index)
PUT datastream-000001
{
"aliases": {
"datastream": {
"is_write_index": true
}
}
}
创建一个初始索引,并设置索引可写入。
配置完成,下面进行验证过程。
4,配置lifecycle检测时间
默认十分钟,便于测试,改为10s。
5,验证
POST datastream/_doc
{
"message":"hello world"
}
通常对外暴露别名,所以索引采用别名写入数据,
因为,别名”datastream“匹配index_patterns,写入后,es根据配置的策略“datastream_policy”进行rollover。
用以下命令查看索引rollover进度。
等待30秒后,返回的数据如下:
{
"indices" : {
"datastream-000002" : {
"index" : "datastream-000002",
"managed" : true,
"policy" : "datastream_policy",
"lifecycle_date_millis" : 1596442291089,
"age" : "43.77s",
"phase" : "hot",
"phase_time_millis" : 1596442292420,
"action" : "rollover",
"action_time_millis" : 1596442301555,
"step" : "check-rollover-ready",
"step_time_millis" : 1596442301555,
"phase_execution" : {
"policy" : "datastream_policy",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_docs" : 1
}
}
},
"version" : 1,
"modified_date_in_millis" : 1596441845846
}
},
"datastream-000001" : {
"index" : "datastream-000001",
"managed" : true,
"policy" : "datastream_policy",
"lifecycle_date_millis" : 1596442291025,
"age" : "43.83s",
"phase" : "delete",
"phase_time_millis" : 1596442330859,
"action" : "delete",
"action_time_millis" : 1596442330859,
"step" : "wait-for-shard-history-leases",
"step_time_millis" : 1596442330859,
"phase_execution" : {
"policy" : "datastream_policy",
"phase_definition" : {
"min_age" : "30s",
"actions" : {
"delete" : { }
}
},
"version" : 1,
"modified_date_in_millis" : 1596441845846
}
}
}
}10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.37.38.39.40.41.42.43.44.45.46.47.48.49.50.51.52.53.54.
索引自动创建了一个新的"datastream-000002",旧的索引”datastream-000001“ action变成DELETE。
再过一段时间再次查看索引
结果如下:
{
"indices" : {
"datastream-000002" : {
"index" : "datastream-000002",
"managed" : true,
"policy" : "datastream_policy",
"lifecycle_date_millis" : 1596442291089,
"age" : "1.45m",
"phase" : "hot",
"phase_time_millis" : 1596442292420,
"action" : "rollover",
"action_time_millis" : 1596442301555,
"step" : "check-rollover-ready",
"step_time_millis" : 1596442301555,
"phase_execution" : {
"policy" : "datastream_policy",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_docs" : 1
}
}
},
"version" : 1,
"modified_date_in_millis" : 1596441845846
}
}
}
}2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.
旧的索引已经删除,在做日志查询时,我们只需要使用别名”datastream“进行日志查询即可,es便可无感知完成日志存储删除动作。
es可以用来存储日志,一般日志存储只是短期保存,超过一定时间日志要是能自动删除最好,这样保证索引文档不会过多,查询时效性也能得到保证。
本文参考的官网地址是: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/getting-started-index-lifecycle-management.html 。
索引的生命周期分为四个阶段:HOT->WARM->COLD->DELETE。
上面除了HOT为必须的阶段外,其他为非必须阶段,可以任意选择配置。因为日志索引只要满足自己删除功能,所以下文只配置了HOT与DELETE阶段。
三步实现完成es生命周期管理
配置策略(policy)、索引模版(template)、索引(index)
1,配置策略(policy)
PUT _ilm/policy/datastream_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover":{
"max_docs":1
}
}
},
"delete": {
"min_age": "30s",
"actions": {
"delete": {}
}
}
}
}
}1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.
创建了一个 “datastream_policy”策略,策略含义是:当hot阶段索引文档超过一条,就会发生rollover——便于测试,
delete阶段,超过30s执行删除动作。
还可以选择其它维度控制,索引大小,索引时间:
2,索引模版(template)
PUT _template/datastream_template
{
"index_patterns": ["datastream-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "datastream_policy",
"index.lifecycle.rollover_alias": "datastream"
}
}2.3.4.5.6.7.8.9.10.
创建了一个索引模板,"index_patterns": ["datastream-*"],索引以datastream-开头的则自动采用setting的配置;
"index.lifecycle.name": "datastream_policy", 采用”datastream_policy“的策略;
"index.lifecycle.rollover_alias": "datastream",创建使用该模版创建的索引统一用”datastream“的别名进行管理
3,索引(index)
PUT datastream-000001
{
"aliases": {
"datastream": {
"is_write_index": true
}
}
}3.4.6.7.8.
创建一个初始索引,并设置索引可写入。
配置完成,下面进行验证过程。
4,配置lifecycle检测时间
默认十分钟,便于测试,改为10s。
5,验证
POST datastream/_doc
{
"message":"hello world"
}
通常对外暴露别名,所以索引采用别名写入数据,
因为,别名”datastream“匹配index_patterns,写入后,es根据配置的策略“datastream_policy”进行rollover。
用以下命令查看索引rollover进度。
等待30秒后,返回的数据如下:
{
"indices" : {
"datastream-000002" : {
"index" : "datastream-000002",
"managed" : true,
"policy" : "datastream_policy",
"lifecycle_date_millis" : 1596442291089,
"age" : "43.77s",
"phase" : "hot",
"phase_time_millis" : 1596442292420,
"action" : "rollover",
"action_time_millis" : 1596442301555,
"step" : "check-rollover-ready",
"step_time_millis" : 1596442301555,
"phase_execution" : {
"policy" : "datastream_policy",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_docs" : 1
}
}
},
"version" : 1,
"modified_date_in_millis" : 1596441845846
}
},
"datastream-000001" : {
"index" : "datastream-000001",
"managed" : true,
"policy" : "datastream_policy",
"lifecycle_date_millis" : 1596442291025,
"age" : "43.83s",
"phase" : "delete",
"phase_time_millis" : 1596442330859,
"action" : "delete",
"action_time_millis" : 1596442330859,
"step" : "wait-for-shard-history-leases",
"step_time_millis" : 1596442330859,
"phase_execution" : {
"policy" : "datastream_policy",
"phase_definition" : {
"min_age" : "30s",
"actions" : {
"delete" : { }
}
},
"version" : 1,
"modified_date_in_millis" : 1596441845846
}
}
}
}10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.37.38.39.40.41.42.43.44.45.46.47.48.49.50.51.52.53.54.
索引自动创建了一个新的"datastream-000002",旧的索引”datastream-000001“ action变成DELETE。
再过一段时间再次查看索引
结果如下:
{
"indices" : {
"datastream-000002" : {
"index" : "datastream-000002",
"managed" : true,
"policy" : "datastream_policy",
"lifecycle_date_millis" : 1596442291089,
"age" : "1.45m",
"phase" : "hot",
"phase_time_millis" : 1596442292420,
"action" : "rollover",
"action_time_millis" : 1596442301555,
"step" : "check-rollover-ready",
"step_time_millis" : 1596442301555,
"phase_execution" : {
"policy" : "datastream_policy",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_docs" : 1
}
}
},
"version" : 1,
"modified_date_in_millis" : 1596441845846
}
}
}
}10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.
旧的索引已经删除,在做日志查询时,我们只需要使用别名”datastream“进行日志查询即可,es便可无感知完成日志存储删除动作。
Loading Comments...