101-150 of 10000 results (27ms)
2026-11-20 §
15:53 <bking@deploy2003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply [production]
15:53 <bking@deploy2003> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [production]
15:52 <bking@deploy2003> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [production]
15:52 <inflatador> bking@deploy2003 `charlie --services_dir dse-k8s-services -s opensearch-* -e dse-k8s-* apply` T445272 [production]
15:52 <bking@deploy2003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [production]
15:52 <bking@deploy2003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [production]
15:28 <krinkle@deploy2003> Finished scap sync-world: Backport for [[gerrit:1370680|MediaWikiEntryPoint: Introduce unsampled post.log (T445147)]] (1.48.0-wmf.5) (duration: 10m 48s) [production]
15:22 <krinkle@deploy2003> krinkle: Continuing with deployment [production]
15:20 <krinkle@deploy2003> krinkle: Backport for [[gerrit:1370680|MediaWikiEntryPoint: Introduce unsampled post.log (T445147)]] (1.48.0-wmf.5) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
15:18 <krinkle@deploy2003> Started scap sync-world: Backport for [[gerrit:1370680|MediaWikiEntryPoint: Introduce unsampled post.log (T445147)]] (1.48.0-wmf.5) [production]
14:49 <urandom> forcing major compaction, sessionstore1004-a (diagnostic) — T445661 [production]
14:40 <javiermonton@deploy2003> Finished scap deploy-service mw-page-html-content-change-enrich: scaling down to its regular number of replicas after incident. (duration: 01m 59s) [production]
14:38 <javiermonton@deploy2003> scap deploy-service mw-page-html-content-change-enrich: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying mw-page-html-content-change-enrich [production]
14:38 <javiermonton@deploy2003> Started scap deploy-service mw-page-html-content-change-enrich: scaling down to its regular number of replicas after incident. [production]
14:37 <javiermonton@deploy2003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply [production]
14:37 <javiermonton@deploy2003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply [production]
14:27 <ayounsi@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [production]
14:23 <ayounsi@cumin1004> START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [production]
14:22 <ayounsi@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [production]
14:18 <ayounsi@cumin1004> START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [production]
13:42 <ayounsi@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [production]
13:38 <ayounsi@cumin1004> START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [production]
13:34 <ayounsi@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [production]
13:30 <ayounsi@cumin1004> START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [production]
13:25 <ammarpad@deploy2003> mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wiki Loves Children' 'Wiki Loves Childhood' Ammarpad --reason 'Requested at [[:phab:T445242|T445242]]' [production]
13:20 <marostegui@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1210.eqiad.wmnet with reason: Maintenance [production]
13:19 <ammarpad@deploy2003> mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=itwiki --logwiki=metawiki 'Nikolay Omonov' User3761 # T445620 [production]
13:02 <marostegui@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [production]
13:02 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1275 (T445307)', diff saved to https://phabricator.wikimedia.org/P97664 and previous config saved to /var/cache/conftool/dbconfig/20261120-130242-marostegui.json [production]
12:47 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1275', diff saved to https://phabricator.wikimedia.org/P97663 and previous config saved to /var/cache/conftool/dbconfig/20261120-124740-marostegui.json [production]
12:37 <arnaudb@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on etherpad2002.codfw.wmnet,etherpad1004.eqiad.wmnet with reason: maintenance [production]
12:32 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1275', diff saved to https://phabricator.wikimedia.org/P97662 and previous config saved to /var/cache/conftool/dbconfig/20261120-123238-marostegui.json [production]
12:17 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1275 (T445307)', diff saved to https://phabricator.wikimedia.org/P97661 and previous config saved to /var/cache/conftool/dbconfig/20261120-121736-marostegui.json [production]
12:14 <marostegui@cumin1004> dbctl commit (dc=all): 'Depooling db1275 (T445307)', diff saved to https://phabricator.wikimedia.org/P97660 and previous config saved to /var/cache/conftool/dbconfig/20261120-121426-marostegui.json [production]
12:14 <marostegui@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 6 hosts with reason: Maintenance [production]
12:14 <marostegui@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1275.eqiad.wmnet with reason: Maintenance [production]
12:13 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1274 (T445307)', diff saved to https://phabricator.wikimedia.org/P97659 and previous config saved to /var/cache/conftool/dbconfig/20261120-121357-marostegui.json [production]
12:00 <root@deploy2003> helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [production]
12:00 <root@deploy2003> helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [production]
12:00 <root@deploy2003> helmfile [staging] DONE helmfile.d/services/shellbox: sync [production]
12:00 <root@deploy2003> helmfile [staging] START helmfile.d/services/shellbox: sync [production]
11:58 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1274', diff saved to https://phabricator.wikimedia.org/P97658 and previous config saved to /var/cache/conftool/dbconfig/20261120-115855-marostegui.json [production]
11:52 <awight@deploy2003> helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply [production]
11:52 <awight@deploy2003> helmfile [codfw] START helmfile.d/services/mw-experimental: apply [production]
11:43 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1274', diff saved to https://phabricator.wikimedia.org/P97657 and previous config saved to /var/cache/conftool/dbconfig/20261120-114353-marostegui.json [production]
11:28 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1274 (T445307)', diff saved to https://phabricator.wikimedia.org/P97656 and previous config saved to /var/cache/conftool/dbconfig/20261120-112851-marostegui.json [production]
11:25 <marostegui@cumin1004> dbctl commit (dc=all): 'Depooling db1274 (T445307)', diff saved to https://phabricator.wikimedia.org/P97655 and previous config saved to /var/cache/conftool/dbconfig/20261120-112546-marostegui.json [production]
11:25 <marostegui@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1274.eqiad.wmnet with reason: Maintenance [production]
11:25 <marostegui@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1230 (T445307)', diff saved to https://phabricator.wikimedia.org/P97654 and previous config saved to /var/cache/conftool/dbconfig/20261120-112529-marostegui.json [production]
11:11 <marostegui@cumin1004> DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: down [production]