2651-2700 of 10000 results (59ms)
2022-01-31 §
07:50 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti1010.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage [production]
07:39 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2075.codfw.wmnet with OS bullseye [production]
07:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P19587 and previous config saved to /var/cache/conftool/dbconfig/20220131-073754-marostegui.json [production]
07:33 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2111.codfw.wmnet with OS bullseye [production]
07:32 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2128.codfw.wmnet with OS bullseye [production]
07:29 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2113.codfw.wmnet with OS bullseye [production]
07:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2137.codfw.wmnet with OS bullseye [production]
07:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298559)', diff saved to https://phabricator.wikimedia.org/P19586 and previous config saved to /var/cache/conftool/dbconfig/20220131-072249-marostegui.json [production]
07:20 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 100%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19585 and previous config saved to /var/cache/conftool/dbconfig/20220131-071959-root.json [production]
07:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 100%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19584 and previous config saved to /var/cache/conftool/dbconfig/20220131-071948-root.json [production]
07:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1112 (T298559)', diff saved to https://phabricator.wikimedia.org/P19583 and previous config saved to /var/cache/conftool/dbconfig/20220131-071350-marostegui.json [production]
07:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
07:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
07:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
07:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
07:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
07:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
07:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 75%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19582 and previous config saved to /var/cache/conftool/dbconfig/20220131-070456-root.json [production]
07:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 75%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19581 and previous config saved to /var/cache/conftool/dbconfig/20220131-070444-root.json [production]
06:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
06:57 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
06:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123 (T298559)', diff saved to https://phabricator.wikimedia.org/P19580 and previous config saved to /var/cache/conftool/dbconfig/20220131-065733-marostegui.json [production]
06:55 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2128.codfw.wmnet with OS bullseye [production]
06:54 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2137.codfw.wmnet with OS bullseye [production]
06:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 50%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19579 and previous config saved to /var/cache/conftool/dbconfig/20220131-064952-root.json [production]
06:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 50%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19578 and previous config saved to /var/cache/conftool/dbconfig/20220131-064941-root.json [production]
06:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123', diff saved to https://phabricator.wikimedia.org/P19577 and previous config saved to /var/cache/conftool/dbconfig/20220131-064228-marostegui.json [production]
06:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 25%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19576 and previous config saved to /var/cache/conftool/dbconfig/20220131-063448-root.json [production]
06:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 25%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19575 and previous config saved to /var/cache/conftool/dbconfig/20220131-063437-root.json [production]
06:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1113.eqiad.wmnet with OS bullseye [production]
06:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123', diff saved to https://phabricator.wikimedia.org/P19574 and previous config saved to /var/cache/conftool/dbconfig/20220131-062723-marostegui.json [production]
06:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123 (T298559)', diff saved to https://phabricator.wikimedia.org/P19573 and previous config saved to /var/cache/conftool/dbconfig/20220131-061219-marostegui.json [production]
06:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove logpager from s4 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P19572 and previous config saved to /var/cache/conftool/dbconfig/20220131-061121-marostegui.json [production]
06:04 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1113.eqiad.wmnet with OS bullseye [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1123 (T298559)', diff saved to https://phabricator.wikimedia.org/P19571 and previous config saved to /var/cache/conftool/dbconfig/20220131-060326-marostegui.json [production]
06:03 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
06:03 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
05:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1113 (s5,s6) T299479', diff saved to https://phabricator.wikimedia.org/P19570 and previous config saved to /var/cache/conftool/dbconfig/20220131-055947-marostegui.json [production]
05:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
05:55 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
2022-01-29 §
21:08 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices2003-dev.wikimedia.org with OS bullseye [production]
18:04 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudservices2003-dev.wikimedia.org with OS bullseye [production]
17:57 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol2004-dev.wikimedia.org with OS bullseye [production]
16:56 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcontrol2004-dev.wikimedia.org with OS bullseye [production]
13:53 <hashar> contint1001 and contint2001 : pruning old reflog from Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -maxdepth 4 -type d -name .git -print -execdir git reflog expire --all --expire=now \;` [production]
05:25 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcontrol2003-dev.wikimedia.org with OS bullseye [production]
04:45 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcontrol2003-dev.wikimedia.org with OS bullseye [production]
00:14 <ebernhardson> restart elasticsearch_6@production-search-psi-eqiad on elastic1049 to address CirrusSearchJVMGCOldPoolFlatlined alert [production]
2022-01-28 §
21:52 <mutante> purging font packages from mwdebug* and scandium* T294378 [production]
21:47 <mutante> purging font packages from remaining appservers in codfw mw23* ranges.. T294378 [production]