2022-07-19
§
|
09:40 |
<ayounsi@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts netbox1001.wikimedia.org |
[production] |
09:38 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netbox2001.wikimedia.org |
[production] |
09:38 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
09:37 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
09:36 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 10:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
09:34 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
09:29 |
<ayounsi@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts netbox2001.wikimedia.org |
[production] |
09:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
09:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
09:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
09:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
09:00 |
<urbanecm> |
Deployed patch for T313205 |
[production] |
08:58 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
08:58 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
08:58 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
08:57 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
08:39 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2018.codfw.wmnet to cluster codfw and group D |
[production] |
08:22 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti2018.codfw.wmnet to cluster codfw and group D |
[production] |
08:15 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2018.codfw.wmnet |
[production] |
08:05 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2018.codfw.wmnet |
[production] |
07:56 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance |
[production] |
07:55 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance |
[production] |
07:55 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 8 hosts with reason: Maintenance |
[production] |
07:55 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 8 hosts with reason: Maintenance |
[production] |
07:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Adjust db2167:3311 and db2167:3318 weight T311493', diff saved to https://phabricator.wikimedia.org/P31390 and previous config saved to /var/cache/conftool/dbconfig/20220719-071836-marostegui.json |
[production] |
07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db2167:3311 and db2167:3318 to dbctl T311493', diff saved to https://phabricator.wikimedia.org/P31389 and previous config saved to /var/cache/conftool/dbconfig/20220719-071656-marostegui.json |
[production] |
06:58 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2084.codfw.wmnet |
[production] |
06:56 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
06:51 |
<marostegui@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
06:47 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db2084.codfw.wmnet |
[production] |
05:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db2084 from dbctl T313121', diff saved to https://phabricator.wikimedia.org/P31386 and previous config saved to /var/cache/conftool/dbconfig/20220719-051725-marostegui.json |
[production] |
02:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
02:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
02:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
02:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
02:06 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
02:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
02:05 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
02:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
2022-07-18
§
|
23:58 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1050.eqiad.wmnet |
[production] |
23:46 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cloudvirt1050.eqiad.wmnet |
[production] |
23:19 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudvirt1049.eqiad.wmnet |
[production] |
23:07 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cloudvirt1049.eqiad.wmnet |
[production] |
21:46 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
21:45 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
21:45 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
21:42 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
21:36 |
<sbassett> |
Deployed security fix for T309894 |
[production] |
20:58 |
<ebernhardson> |
start reindex of all wikis except commonswiki and wikidatawiki in eqiad and codfw cirrus clusters |
[production] |
20:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |