951-1000 of 10000 results (44ms)
2022-01-17 §
10:17 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1152.eqiad.wmnet with OS bullseye [production]
10:15 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1153.eqiad.wmnet with OS bullseye [production]
10:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P18758 and previous config saved to /var/cache/conftool/dbconfig/20220117-101450-marostegui.json [production]
10:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2144.codfw.wmnet with OS bullseye [production]
10:04 <moritzm> switching kubetcd1004 to DRBD-backed storage (required for ganeti update) [production]
10:03 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubetcd1004.eqiad.wmnet with reason: switch to drbd storage [production]
10:03 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on kubetcd1004.eqiad.wmnet with reason: switch to drbd storage [production]
10:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2143.codfw.wmnet with OS bullseye [production]
09:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131 (T285149)', diff saved to https://phabricator.wikimedia.org/P18757 and previous config saved to /var/cache/conftool/dbconfig/20220117-095945-marostegui.json [production]
09:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1131 (T285149)', diff saved to https://phabricator.wikimedia.org/P18756 and previous config saved to /var/cache/conftool/dbconfig/20220117-095837-marostegui.json [production]
09:58 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [production]
09:58 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [production]
09:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18755 and previous config saved to /var/cache/conftool/dbconfig/20220117-095830-marostegui.json [production]
09:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P18754 and previous config saved to /var/cache/conftool/dbconfig/20220117-094325-marostegui.json [production]
09:30 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2144.codfw.wmnet with OS bullseye [production]
09:30 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2143.codfw.wmnet with OS bullseye [production]
09:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P18753 and previous config saved to /var/cache/conftool/dbconfig/20220117-092820-marostegui.json [production]
09:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1017.eqiad.wmnet with OS bullseye [production]
09:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18752 and previous config saved to /var/cache/conftool/dbconfig/20220117-091316-marostegui.json [production]
09:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1113:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18751 and previous config saved to /var/cache/conftool/dbconfig/20220117-091308-marostegui.json [production]
09:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
09:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
09:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18750 and previous config saved to /var/cache/conftool/dbconfig/20220117-091300-marostegui.json [production]
08:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18749 and previous config saved to /var/cache/conftool/dbconfig/20220117-085756-marostegui.json [production]
08:53 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1017.eqiad.wmnet with OS bullseye [production]
08:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18748 and previous config saved to /var/cache/conftool/dbconfig/20220117-084251-marostegui.json [production]
08:36 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM schema1003.eqiad.wmnet [production]
08:34 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM schema1003.eqiad.wmnet [production]
08:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18747 and previous config saved to /var/cache/conftool/dbconfig/20220117-082746-marostegui.json [production]
08:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18746 and previous config saved to /var/cache/conftool/dbconfig/20220117-082638-marostegui.json [production]
08:26 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
08:26 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
08:21 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM schema1004.eqiad.wmnet [production]
08:17 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM schema1004.eqiad.wmnet [production]
06:59 <elukey> `systemctl reset-failed ifup@ens5.service` on an-test-client1001 and kafka-test1010 [production]
06:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1016.eqiad.wmnet with OS bullseye [production]
05:57 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1016.eqiad.wmnet with OS bullseye [production]
2022-01-16 §
08:21 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: sync on production [production]
08:20 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply on staging [production]
08:20 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply on production [production]
08:18 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: sync on production [production]
08:17 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply on staging [production]
08:17 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply on production [production]
2022-01-15 §
08:55 <legoktm> finished running recountCategories on s4 wikis (T299244) [production]
07:58 <legoktm> finished running recountCategories on s7 wikis (T299244) [production]
07:51 <legoktm> finished running recountCategories on s2 wikis (T299244) [production]
06:41 <<legoktm>> finished running recountCategories on s3 wikis (T299244) [production]
06:21 <<legoktm>> finished running recountCategories on s6 wikis (T299244) [production]
06:19 <<legoktm>> finished running recountCategories on s5 wikis (T299244) [production]
06:18 <<legoktm>> finished running recountCategories on s8 wikis (T299244) [production]