4551-4600 of 10000 results (66ms)
2022-06-15 ยง
17:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:03 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.16 refs T308069 [production]
16:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P29848 and previous config saved to /var/cache/conftool/dbconfig/20220615-165727-marostegui.json [production]
16:54 <brennen> train 1.39.0-wmf.16 (T308069): no current blockers - rolling to group0 [production]
16:44 <jynus> reestarting replication for m3 on db1117, not db2078 [production]
16:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118 (T310011)', diff saved to https://phabricator.wikimedia.org/P29847 and previous config saved to /var/cache/conftool/dbconfig/20220615-164222-marostegui.json [production]
16:37 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host an-presto1012.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:31 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host an-presto1011.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:30 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host an-presto1010.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:30 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host an-presto1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:30 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-presto1007.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:30 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-presto1008.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:30 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-presto1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:29 <brennen> phabricator upgrade finished [production]
16:27 <krinkle@deploy1002> Synchronized multiversion/: Id8cdb8aef70f6672 (duration: 03m 41s) [production]
16:25 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
16:24 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
16:24 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
16:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
16:21 <pt1979@cumin1001> END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host backup1009.eqiad.wmnet [production]
16:21 <pt1979@cumin1001> START - Cookbook sre.hosts.dhcp for host backup1009.eqiad.wmnet [production]
16:13 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host an-presto1008.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:12 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:12 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host an-presto1007.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:11 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host an-presto1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
16:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1118 (T310011)', diff saved to https://phabricator.wikimedia.org/P29845 and previous config saved to /var/cache/conftool/dbconfig/20220615-160838-marostegui.json [production]
16:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
16:08 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
16:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128 (T310011)', diff saved to https://phabricator.wikimedia.org/P29844 and previous config saved to /var/cache/conftool/dbconfig/20220615-160830-marostegui.json [production]
16:08 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS buster [production]
15:56 <otto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply [production]
15:55 <otto@deploy1002> helmfile [eqiad] START helmfile.d/services/eventstreams: apply [production]
15:55 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [production]
15:55 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventstreams: apply [production]
15:53 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [production]
15:53 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet [production]
15:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P29843 and previous config saved to /var/cache/conftool/dbconfig/20220615-155325-marostegui.json [production]
15:53 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventstreams: apply [production]
15:51 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [production]
15:51 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventstreams: apply [production]
15:50 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventstreams: apply [production]
15:49 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventstreams: apply [production]
15:49 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet [production]
15:40 <mutante> phabricator upgrade in progress [production]
15:39 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventstreams: apply [production]
15:39 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventstreams: apply [production]
15:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20220615-153820-marostegui.json [production]
15:35 <brennen> starting phabricator deploy, momentary downtime expected while Apache restarts and migrations run [production]