4301-4350 of 10000 results (87ms)
2022-06-16 ยง
18:54 <dzahn@cumin2002> START - Cookbook sre.hosts.decommission for hosts gitlab-runner1001.eqiad.wmnet [production]
18:53 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts gitlab-runner1001.eqiad.wmnet [production]
18:53 <dzahn@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
18:53 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:50 <dzahn@cumin2002> START - Cookbook sre.dns.netbox [production]
18:49 <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.16 refs T308069 [production]
18:48 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
18:44 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
18:44 <brennen> train 1.39.0-wmf.16 (T308069): no current blockers - rolling to all wikis [production]
18:44 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:42 <brennen@deploy1002> Synchronized php-1.39.0-wmf.16/extensions/CheckUser/src/Hooks.php: Backport: [[gerrit:806246|Only try to create User object if username is not null (T310747)]] (duration: 03m 23s) [production]
18:40 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P29903 and previous config saved to /var/cache/conftool/dbconfig/20220616-184015-marostegui.json [production]
18:29 <dzahn@cumin2002> START - Cookbook sre.hosts.decommission for hosts gitlab-runner1001.eqiad.wmnet [production]
18:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P29902 and previous config saved to /var/cache/conftool/dbconfig/20220616-182510-marostegui.json [production]
18:13 <btullis@deploy1002> helmfile [codfw] DONE helmfile.d/services/datahub: sync on main [production]
18:12 <btullis@deploy1002> helmfile [codfw] START helmfile.d/services/datahub: sync on main [production]
18:12 <btullis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main [production]
18:11 <btullis@deploy1002> helmfile [eqiad] START helmfile.d/services/datahub: sync on main [production]
18:10 <btullis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main [production]
18:10 <btullis@deploy1002> helmfile [eqiad] START helmfile.d/services/datahub: sync on main [production]
18:10 <btullis@deploy1002> helmfile [codfw] DONE helmfile.d/services/datahub: sync on main [production]
18:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158 (T302659)', diff saved to https://phabricator.wikimedia.org/P29901 and previous config saved to /var/cache/conftool/dbconfig/20220616-181005-marostegui.json [production]
18:10 <btullis@deploy1002> helmfile [codfw] START helmfile.d/services/datahub: sync on main [production]
17:59 <brennen> end of phabricator deploy [production]
17:46 <brennen> starting phabricator deploy, momentary downtime expected while services restart [production]
17:42 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab.wmfusercontent.org with reason: bug fix [production]
17:42 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on phab.wmfusercontent.org with reason: bug fix [production]
17:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1158 (T302659)', diff saved to https://phabricator.wikimedia.org/P29900 and previous config saved to /var/cache/conftool/dbconfig/20220616-173738-marostegui.json [production]
17:37 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
17:37 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
17:37 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
17:37 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
17:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174 (T302659)', diff saved to https://phabricator.wikimedia.org/P29899 and previous config saved to /var/cache/conftool/dbconfig/20220616-173725-marostegui.json [production]
17:31 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab1001.eqiad.wmnet with reason: bug fix [production]
17:31 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on phab1001.eqiad.wmnet with reason: bug fix [production]
17:27 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phabricator.wikimedia.org with reason: bug fix [production]
17:27 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on phabricator.wikimedia.org with reason: bug fix [production]
17:26 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx1001.wikimedia.org with reason: New Kernel [production]
17:26 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on mx1001.wikimedia.org with reason: New Kernel [production]
17:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P29898 and previous config saved to /var/cache/conftool/dbconfig/20220616-172220-marostegui.json [production]
17:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P29897 and previous config saved to /var/cache/conftool/dbconfig/20220616-170715-marostegui.json [production]
16:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174 (T302659)', diff saved to https://phabricator.wikimedia.org/P29896 and previous config saved to /var/cache/conftool/dbconfig/20220616-165210-marostegui.json [production]
16:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1174 (T302659)', diff saved to https://phabricator.wikimedia.org/P29895 and previous config saved to /var/cache/conftool/dbconfig/20220616-161844-marostegui.json [production]
16:18 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance [production]
16:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance [production]
16:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181 (T302659)', diff saved to https://phabricator.wikimedia.org/P29894 and previous config saved to /var/cache/conftool/dbconfig/20220616-161835-marostegui.json [production]
16:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P29893 and previous config saved to /var/cache/conftool/dbconfig/20220616-160330-marostegui.json [production]
15:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P29892 and previous config saved to /var/cache/conftool/dbconfig/20220616-154825-marostegui.json [production]
15:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181 (T302659)', diff saved to https://phabricator.wikimedia.org/P29891 and previous config saved to /var/cache/conftool/dbconfig/20220616-153320-marostegui.json [production]