2401-2450 of 10000 results (60ms)
2022-07-19 §
09:40 <ayounsi@cumin1001> START - Cookbook sre.hosts.decommission for hosts netbox1001.wikimedia.org [production]
09:38 <ayounsi@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netbox2001.wikimedia.org [production]
09:38 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:37 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
09:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
09:34 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
09:29 <ayounsi@cumin1001> START - Cookbook sre.hosts.decommission for hosts netbox2001.wikimedia.org [production]
09:05 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
09:04 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
09:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
09:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
09:00 <urbanecm> Deployed patch for T313205 [production]
08:58 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
08:58 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
08:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
08:57 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
08:39 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2018.codfw.wmnet to cluster codfw and group D [production]
08:22 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti2018.codfw.wmnet to cluster codfw and group D [production]
08:15 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2018.codfw.wmnet [production]
08:05 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2018.codfw.wmnet [production]
07:56 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
07:55 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
07:55 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 8 hosts with reason: Maintenance [production]
07:55 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 8 hosts with reason: Maintenance [production]
07:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust db2167:3311 and db2167:3318 weight T311493', diff saved to https://phabricator.wikimedia.org/P31390 and previous config saved to /var/cache/conftool/dbconfig/20220719-071836-marostegui.json [production]
07:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db2167:3311 and db2167:3318 to dbctl T311493', diff saved to https://phabricator.wikimedia.org/P31389 and previous config saved to /var/cache/conftool/dbconfig/20220719-071656-marostegui.json [production]
06:58 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2084.codfw.wmnet [production]
06:56 <marostegui@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:51 <marostegui@cumin1001> START - Cookbook sre.dns.netbox [production]
06:47 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db2084.codfw.wmnet [production]
05:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db2084 from dbctl T313121', diff saved to https://phabricator.wikimedia.org/P31386 and previous config saved to /var/cache/conftool/dbconfig/20220719-051725-marostegui.json [production]
02:32 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
02:06 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:05 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
2022-07-18 §
23:58 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1050.eqiad.wmnet [production]
23:46 <andrew@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1050.eqiad.wmnet [production]
23:19 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudvirt1049.eqiad.wmnet [production]
23:07 <andrew@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1049.eqiad.wmnet [production]
21:46 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:45 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:45 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:36 <sbassett> Deployed security fix for T309894 [production]
20:58 <ebernhardson> start reindex of all wikis except commonswiki and wikidatawiki in eqiad and codfw cirrus clusters [production]
20:47 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]