GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||
GeezerVison is showing phabricator.wikimedia.org — T305175 Fix conflict between monthly and weekly index buckets
Assigned To Authored By
Project Tags Referenced Files None Subscribers
Description Indexes created with monthly indexes %{+YYYY.MM} share the same pattern as weekly indexes %{+xxxx.ww}. This could lead to problems when curating weekly and monthly indexes if they matching the same curator pattern. One possible solution is to make weekly buckets underscore-separated and calendar year buckets dot-separated. @herron @fgiunchedi what do you think about that convention?
Details Related Changes in Gerrit:
Related Objects Event TimelineRestricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 31 2022, 4:43 PM2022-03-31 16:43:35 (UTC+0) Comment Actions Sounds like that'd work, although I wonder if there are any alternatives that may be more obvious at a glance? Could we get away with doing something like including the unit as a keyword before the stamp? e.g. ecs-1.7.0-5-alerts-bucket-daily-2022.01.01 Comment Actions Agreed, something explicit in the index name seems preferable to me too and less prone to errors. Comment Actions
I want to explore this idea further. It would remove the need to key off of outputs by default we included bucket in the index pattern. This would look like:
This matches ^(?<output>[a-z0-9]+)-((?<partition>[a-z0-9]+)-)?((?<version>[0-9\.]+)-(?<revision>[0-9]+)-)?(?<bucket>daily|weekly|monthly|yearly)-(?<created>[0-9.]+)$. I do not think this will simplify our curator config much. We'll be trading output-defined actions for a set of implicit (bucket-defined) actions and adding exceptions. This would look like:
This introduces a problem where if we introduce a new monthly index with a 9-month retention (e.g. ecs-foo-*-monthly-2022.04), the monthly action (delete older than 3 months) would have to exclude the pattern in addition to setting up the 9-month action. I see a link between the creation format and retention. If we were to extend the definition of "bucket" to combine format and retention ("policy", perhaps?), we could reuse curator actions more effectively. An example strawman approach: ^(?<output>[a-z0-9]+)-((?<partition>[a-z0-9]+)-)?((?<version>[0-9\.]+)-(?<revision>[0-9]+)-)?(?<policy>[a-z0-9]+)-(?<created>[0-9.]+)$ daily: format: %Y.%m.%d retention: 90 days example: logstash-mediawiki-daily-2022.04.04 weekly: format: %Y.%W retention: 14 weeks example: w3creportingapi-1.0.0-2-weekly-2022.14 monthly: format: %Y.%m retention: 3 months example: ecs-foo-1.7.0-5-monthly-2022.04 3years: format: %Y retention: 3 years example: ecs-alerts-1.7.0-5-3years-2022 2weeks: format: %Y.%W retention: 2 weeks example: ecs-test-1.7.0-5-2weeks-2022.14 2days: format: %Y.%m.%d retention: 2 days example: dlq-1.0.0-1-2days-2022.04.04 What do you think? Comment Actions If I'm understanding correctly the idea is to have a set of generic curator rules that would automatically set retention based on patterns like "2weeks" or "2days" in the index name? One challenge that comes to mind is transitioning retention time on a given index. Say increasing from 2 weeks to 4 weeks. IIUC we would be on the hook to handle renaming/reindexing of the affected existing indices from 2weeks to 4weeks or weekly shortly after the next index rollover to avoid losing them. Comment Actions
Yes, the pattern combines retention period and bucket. "2days", etc. is an example to illustrate the concept. They could just as easily be more kind-of-data-oriented approach (ecs-alerts-1.7.0-5-alerts-2022.14) or a policy revision approach (ecs-1.7.0-5-alerts-0-2022.14).
Good point. Policies named with such specific terms makes them less flexible. Perhaps kind-of-data-oriented policies is preferable here. To evaluate the procedure for adjustment:
It occurs to me we could also get there by leveraging the output-partition pair to achieve the same goal. For example: ecs-1.7.0-5-alerts-2022.14: format: %Y.%W retention: 105 weeks ecs-1.7.0-5-alerts2-2022: format: %Y retention: 2 years It also occurs to me a combination of output-partition-policy revision would get us there. For example: ecs-1.7.0-5-alerts-0-2022.14 format: %Y.%W retention: 105 weeks ecs-1.7.0-5-alerts-1-2022 format: %Y retention: 2 years The more we explore, the more I see the benefits of rolling out a modified index pattern to all indexes like so: ^(?<output>[a-z0-9]+)-(?<partition>[a-z0-9]+)-(?<policy_revision>[0-9]+)-(?<template_version>[0-9\.]+)-(?<template_revision>[0-9]+)-(?<created>[0-9.]+)$
The indexes would then be formatted like so:
This could homogenize the curator filters into a predictable format and replace all regex filters with prefix filters. Unfortunately, we could not construct any generic "catch-all" rules, but we do not have that even now and is not a hard requirement AFAIK. Comment Actions Per discussion today in the Observability meeting, we agreed to evaluate the proposed index name refactor to:
Comment Actions Change 775375 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: populate target index format and add pipeline diagnostics Comment Actions Change 777874 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: replace all instances of @metadata.partition Comment Actions Change 777880 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: set partition on legacy indexes Comment Actions Change 777882 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: transform human-friendly values to bucket date format Comment Actions Change 777891 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: add target index validation step Comment Actions Change 777874 merged by Cwhite: [operations/puppet@production] logstash: replace all instances of @metadata.partition Comment Actions Change 775375 merged by Cwhite: [operations/puppet@production] logstash: populate target index format and add pipeline diagnostics Comment Actions Change 777880 merged by Cwhite: [operations/puppet@production] logstash: set partition on legacy indexes Comment Actions Change 777882 merged by Cwhite: [operations/puppet@production] logstash: transform rotation frequency values to datestamp format Comment Actions Change 777891 merged by Cwhite: [operations/puppet@production] logstash: add target index validation step colewhite changed the task status from Open to In Progress.May 24 2022, 6:46 PM2022-05-24 18:46:18 (UTC+0) colewhite claimed this task. Comment Actions Change 798974 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] beta-logs: enable pipeline-managed index patterns Comment Actions Change 798974 merged by Cwhite: [operations/puppet@production] beta-logs: enable pipeline-managed index patterns Comment Actions Change 798982 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: curator support new and legacy index patterns Comment Actions Change 799001 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: enable pipeline-managed index patterns Comment Actions Change 798982 merged by Cwhite: [operations/puppet@production] logstash: curator support new and legacy index patterns Comment Actions Change 802873 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/software/ecs@master] add new index pattern format Comment Actions Change 802873 merged by jenkins-bot: [operations/software/ecs@master] add new index pattern format Comment Actions Change 803345 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/software/ecs@master] templates: replace all version instances Comment Actions Change 803345 merged by jenkins-bot: [operations/software/ecs@master] templates: replace all version instances Comment Actions Change 803350 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] add new index pattern to ecs templates Comment Actions Change 803350 merged by Cwhite: [operations/puppet@production] logstash: add new index pattern to ecs templates Comment Actions Change 799001 merged by Cwhite: [operations/puppet@production] logstash: enable pipeline-managed index patterns Comment Actions Deployed! Will be watching it this week for any issues. Last thing to do is migrate w3creportingapi-* and dlq-*. colewhite moved this task from Inbox to Prioritized on the Observability-Logging board.Jul 20 2022, 9:11 PM2022-07-20 21:11:16 (UTC+0) Comment Actions Change 815799 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: add missing closing curly brace Comment Actions Change 815799 merged by Cwhite: [operations/puppet@production] logstash: add missing closing curly brace Comment Actions Change 822444 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: replace legacy routing filters Comment Actions Change 822450 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: use logstash routing for w3creportingapi stream Comment Actions Change 822452 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: update production w3creportingapi guard condition Comment Actions Change 822450 merged by Cwhite: [operations/puppet@production] logstash: use logstash routing for w3creportingapi stream Comment Actions Change 822452 merged by Cwhite: [operations/puppet@production] logstash: update production w3creportingapi guard condition Comment Actions Change 824751 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] beta-logs: dlq use logstash-managed index pattern Comment Actions Change 824752 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: dlq use logstash-managed index pattern Comment Actions Change 824753 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] beta-logs: w3creportingapi to use logstash-managed index pattern Comment Actions Change 824754 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: w3creportingapi to use logstash-managed index pattern Comment Actions Change 824751 merged by Cwhite: [operations/puppet@production] beta-logs: dlq use logstash-managed index pattern Comment Actions Change 824753 merged by Cwhite: [operations/puppet@production] beta-logs: w3creportingapi to use logstash-managed index pattern Comment Actions Change 824752 merged by Cwhite: [operations/puppet@production] logstash: dlq use logstash-managed index pattern Comment Actions Change 826372 had a related patch set uploaded (by Cwhite; author: Cwhite): [operations/puppet@production] logstash: add dlq revision to index pattern Comment Actions Change 826372 merged by Cwhite: [operations/puppet@production] logstash: use puppet dlq version and revision for index pattern Comment Actions Change 824754 merged by Cwhite: [operations/puppet@production] logstash: w3creportingapi to use logstash-managed index pattern Comment Actions dlq and w3creportingapi indexes migrated. This is complete. • lmata moved this task from Inbox to Done on the SRE Observability (FY2022/2023-Q1) board.Jul 18 2023, 5:02 PM2023-07-18 17:02:03 (UTC+0) Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits | |||
| You are visitor 01402520 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |