1151-1200 of 10000 results (99ms)
2025-07-22 §
00:33 <bking@cumin2002> conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad [production]
00:26 <bking@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on cirrussearch1081.eqiad.wmnet with reason: bad cluster node/MSS issue [production]
00:23 <amastilovic@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [production]
00:22 <amastilovic@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [production]
00:09 <ryankemper> stop opensearch on `cirrussearch1081` [production]
00:00 <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
2025-07-21 §
23:56 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1171303|Set categorylinks to read new on remaining s2 and s3 wikis]] (duration: 08m 19s) [production]
23:53 <bking@cumin2002> conftool action : set/pooled=false; selector: dnsdisc=search,name=eqiad [production]
23:51 <zabe@deploy1003> zabe: Continuing with sync [production]
23:50 <zabe@deploy1003> zabe: Backport for [[gerrit:1171303|Set categorylinks to read new on remaining s2 and s3 wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
23:48 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1171303|Set categorylinks to read new on remaining s2 and s3 wikis]] [production]
23:32 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1170371|Set categorylinks to read new on remaining large s2 wikis (T397912)]] (duration: 08m 23s) [production]
23:27 <zabe@deploy1003> zabe: Continuing with sync [production]
23:26 <zabe@deploy1003> zabe: Backport for [[gerrit:1170371|Set categorylinks to read new on remaining large s2 wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
23:24 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1170371|Set categorylinks to read new on remaining large s2 wikis (T397912)]] [production]
23:14 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
23:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T399249)', diff saved to https://phabricator.wikimedia.org/P79549 and previous config saved to /var/cache/conftool/dbconfig/20250721-231417-marostegui.json [production]
22:59 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P79548 and previous config saved to /var/cache/conftool/dbconfig/20250721-225910-marostegui.json [production]
22:44 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P79547 and previous config saved to /var/cache/conftool/dbconfig/20250721-224402-marostegui.json [production]
22:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T399249)', diff saved to https://phabricator.wikimedia.org/P79546 and previous config saved to /var/cache/conftool/dbconfig/20250721-222855-marostegui.json [production]
22:24 <jhathaway@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1012.eqiad.wmnet with OS bookworm [production]
22:22 <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
22:05 <jhathaway@cumin1003> START - Cookbook sre.hosts.reimage for host ml-serve1012.eqiad.wmnet with OS bookworm [production]
22:02 <bking@cumin1002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227 [production]
21:52 <maryum> undeploy security fix for T399627 [production]
21:39 <maryum> deploy security fix for T399662 [production]
21:39 <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227 [production]
21:39 <maryum> deploy securty fix for T399662 [production]
21:38 <bking@cumin2002> conftool action : set/weight=10; selector: name=cirrussearch2061.* [production]
21:38 <bking@cumin2002> conftool action : set/pooled=yes; selector: name=cirrussearch2061.* [production]
21:38 <bking@cumin2002> conftool action : set/pooled=yes; selector: name=cirrussearch2061* [production]
21:34 <jhathaway@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1012.eqiad.wmnet with OS bookworm [production]
21:32 <bking@cumin2002> conftool action : set/pooled=yes; selector: name=cirrussearch2061 [production]
21:32 <bking@cumin2002> conftool action : set/weight=10; selector: name=cirrussearch2061 [production]
21:29 <maryum> deploy security patch for T399627 [production]
21:28 <jhathaway@cumin1003> START - Cookbook sre.hosts.reimage for host ml-serve1012.eqiad.wmnet with OS bookworm [production]
21:25 <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: activate new plugins packages - bking@cumin1002 - T397227 [production]
21:21 <jhathaway@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1012.eqiad.wmnet with OS bookworm [production]
21:21 <jhathaway@cumin1003> START - Cookbook sre.hosts.reimage for host ml-serve1012.eqiad.wmnet with OS bookworm [production]
21:14 <jhathaway@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1012.eqiad.wmnet with OS bookworm [production]
21:14 <jhathaway@cumin1003> START - Cookbook sre.hosts.reimage for host ml-serve1012.eqiad.wmnet with OS bookworm [production]
21:13 <jhathaway@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1012.eqiad.wmnet with OS bookworm [production]
21:13 <eileen> civicrm upgraded from 60b2a914 to 63c1860b [production]
21:05 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1212 (T399249)', diff saved to https://phabricator.wikimedia.org/P79544 and previous config saved to /var/cache/conftool/dbconfig/20250721-210531-marostegui.json [production]
21:05 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
21:05 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1212.eqiad.wmnet with reason: Maintenance [production]
21:05 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T399249)', diff saved to https://phabricator.wikimedia.org/P79542 and previous config saved to /var/cache/conftool/dbconfig/20250721-210501-marostegui.json [production]
21:04 <cjming> end of UTC late backport window [production]
21:04 <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1171251|InstrumentConfigsFetcher: Make updating configs asynchronous (T398422)]] (duration: 46m 25s) [production]
20:55 <jhathaway@cumin1003> START - Cookbook sre.hosts.reimage for host ml-serve1012.eqiad.wmnet with OS bookworm [production]