7101-7150 of 10000 results (61ms)
2022-01-31 §
07:20 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 100%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19585 and previous config saved to /var/cache/conftool/dbconfig/20220131-071959-root.json [production]
07:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 100%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19584 and previous config saved to /var/cache/conftool/dbconfig/20220131-071948-root.json [production]
07:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1112 (T298559)', diff saved to https://phabricator.wikimedia.org/P19583 and previous config saved to /var/cache/conftool/dbconfig/20220131-071350-marostegui.json [production]
07:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
07:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
07:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
07:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
07:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
07:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
07:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 75%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19582 and previous config saved to /var/cache/conftool/dbconfig/20220131-070456-root.json [production]
07:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 75%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19581 and previous config saved to /var/cache/conftool/dbconfig/20220131-070444-root.json [production]
06:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
06:57 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
06:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123 (T298559)', diff saved to https://phabricator.wikimedia.org/P19580 and previous config saved to /var/cache/conftool/dbconfig/20220131-065733-marostegui.json [production]
06:55 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2128.codfw.wmnet with OS bullseye [production]
06:54 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2137.codfw.wmnet with OS bullseye [production]
06:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 50%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19579 and previous config saved to /var/cache/conftool/dbconfig/20220131-064952-root.json [production]
06:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 50%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19578 and previous config saved to /var/cache/conftool/dbconfig/20220131-064941-root.json [production]
06:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123', diff saved to https://phabricator.wikimedia.org/P19577 and previous config saved to /var/cache/conftool/dbconfig/20220131-064228-marostegui.json [production]
06:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 25%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19576 and previous config saved to /var/cache/conftool/dbconfig/20220131-063448-root.json [production]
06:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 25%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19575 and previous config saved to /var/cache/conftool/dbconfig/20220131-063437-root.json [production]
06:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1113.eqiad.wmnet with OS bullseye [production]
06:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123', diff saved to https://phabricator.wikimedia.org/P19574 and previous config saved to /var/cache/conftool/dbconfig/20220131-062723-marostegui.json [production]
06:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123 (T298559)', diff saved to https://phabricator.wikimedia.org/P19573 and previous config saved to /var/cache/conftool/dbconfig/20220131-061219-marostegui.json [production]
06:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove logpager from s4 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P19572 and previous config saved to /var/cache/conftool/dbconfig/20220131-061121-marostegui.json [production]
06:04 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1113.eqiad.wmnet with OS bullseye [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1123 (T298559)', diff saved to https://phabricator.wikimedia.org/P19571 and previous config saved to /var/cache/conftool/dbconfig/20220131-060326-marostegui.json [production]
06:03 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
06:03 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
05:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1113 (s5,s6) T299479', diff saved to https://phabricator.wikimedia.org/P19570 and previous config saved to /var/cache/conftool/dbconfig/20220131-055947-marostegui.json [production]
05:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
05:55 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
2022-01-30 §
14:41 <taavi> created a neutron port with ip 172.16.2.46 for a service ip for toolforge redis automatic failover T278541 [tools]
14:22 <taavi> creating a cluster of 3 bullseye redis hosts for T278541 [tools]
12:35 <wm-bot> <lucaswerkmeister> deployed c1d6a79ed2 (update Odia nongendered adjectives) [tools.lexeme-forms]
2022-01-29 §
23:42 <taavi> deploying https://gerrit.wikimedia.org/r/c/analytics/quarry/web/+/749756/ [quarry]
21:08 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices2003-dev.wikimedia.org with OS bullseye [production]
18:24 <taavi> delete 2 puppet prefixes in a weird state T299750 [admin]
18:04 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudservices2003-dev.wikimedia.org with OS bullseye [production]
17:57 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol2004-dev.wikimedia.org with OS bullseye [production]
16:56 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcontrol2004-dev.wikimedia.org with OS bullseye [production]
13:53 <hashar> contint1001 and contint2001 : pruning old reflog from Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -maxdepth 4 -type d -name .git -print -execdir git reflog expire --all --expire=now \;` [production]
05:25 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcontrol2003-dev.wikimedia.org with OS bullseye [production]
04:45 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcontrol2003-dev.wikimedia.org with OS bullseye [production]
00:14 <ebernhardson> restart elasticsearch_6@production-search-psi-eqiad on elastic1049 to address CirrusSearchJVMGCOldPoolFlatlined alert [production]
2022-01-28 §
21:52 <mutante> purging font packages from mwdebug* and scandium* T294378 [production]
21:47 <mutante> purging font packages from remaining appservers in codfw mw23* ranges.. T294378 [production]
21:45 <taavi> running recountCategories.php on all beta wikis per T299823#7652496 [releng]
20:44 <wm-bot> <jeanfred> Deploy 13ed7a3 [tools.integraality]
20:14 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.postgresql.postgres-init (exit_code=99) [production]