201-250 of 10000 results (65ms)
2022-10-13 ยง
11:46 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1005.eqiad.wmnet to plain [production]
11:46 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1005.eqiad.wmnet to plain [production]
11:40 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubetcd1005.eqiad.wmnet to drbd [production]
11:37 <ayounsi@deploy1002> Finished deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext and updated wheels (duration: 04m 41s) [production]
11:33 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P35461 and previous config saved to /var/cache/conftool/dbconfig/20221013-113320-ladsgroup.json [production]
11:33 <ayounsi@deploy1002> Started deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext and updated wheels [production]
11:31 <ayounsi@deploy1002> Finished deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext and updated wheels (duration: 03m 12s) [production]
11:30 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of kubetcd1005.eqiad.wmnet to drbd [production]
11:28 <ayounsi@deploy1002> Started deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext and updated wheels [production]
11:28 <ayounsi@deploy1002> Finished deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext and updated wheels (duration: 03m 34s) [production]
11:27 <Lucas_WMDE> 11:18 ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db2120 (T318950)', diff saved to https://phabricator.wikimedia.org/P35460 and previous config saved to /var/cache/conftool/dbconfig/20221013-111814-ladsgroup.json [production]
11:27 <Lucas_WMDE> 11:15 ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db2120 (T318950)', diff saved to https://phabricator.wikimedia.org/P35458 and previous config saved to /var/cache/conftool/dbconfig/20221013-111556-ladsgroup.json [production]
11:27 <Lucas_WMDE> 11:15 ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance [production]
11:27 <Lucas_WMDE> 11:15 ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance [production]
11:27 <Lucas_WMDE> 11:15 ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db2108 (T318950)', diff saved to https://phabricator.wikimedia.org/P35457 and previous config saved to /var/cache/conftool/dbconfig/20221013-111534-ladsgroup.json [production]
11:26 <Lucas_WMDE> repeating five messages that got missed due to stashbot quit [production]
11:24 <ayounsi@deploy1002> Started deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext and updated wheels [production]
11:24 <ayounsi@deploy1002> Finished deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext and updated wheels (duration: 18m 30s) [production]
11:06 <ayounsi@deploy1002> Started deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext and updated wheels [production]
11:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P35455 and previous config saved to /var/cache/conftool/dbconfig/20221013-110028-ladsgroup.json [production]
10:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P35454 and previous config saved to /var/cache/conftool/dbconfig/20221013-104521-ladsgroup.json [production]
10:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2108 (T318950)', diff saved to https://phabricator.wikimedia.org/P35453 and previous config saved to /var/cache/conftool/dbconfig/20221013-103015-ladsgroup.json [production]
10:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2108 (T318950)', diff saved to https://phabricator.wikimedia.org/P35452 and previous config saved to /var/cache/conftool/dbconfig/20221013-102757-ladsgroup.json [production]
10:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance [production]
10:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance [production]
10:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance [production]
10:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance [production]
10:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
10:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
10:16 <moritzm> draining ganeti1008 T320419 [production]
09:35 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1017.eqiad.wmnet to cluster eqiad and group B [production]
09:33 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1017.eqiad.wmnet to cluster eqiad and group B [production]
09:28 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1017.eqiad.wmnet [production]
09:28 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
09:28 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
09:25 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
09:24 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
09:19 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1017.eqiad.wmnet [production]
09:18 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
09:16 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
09:15 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
09:15 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
09:15 <ayounsi@deploy1002> Finished deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext (duration: 02m 04s) [production]
09:14 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
09:14 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
09:13 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
09:13 <ayounsi@deploy1002> Started deploy [netbox/deploy@7bbf659]: Deploy ntc-netbox-plugin-metrics-ext [production]
09:11 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
08:49 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1017.eqiad.wmnet with OS bullseye [production]
08:48 <vgutierrez> partitioning the ATS cache in cp[2029-2030], cp[6001,6009], cp[1077-1078], cp[5002,5008], cp[3052-3053], cp4022 - T317748 [production]