1251-1300 of 10000 results (67ms)
2022-05-04 §
01:58 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp5002.eqsin.wmnet with OS buster [production]
01:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T307525)', diff saved to https://phabricator.wikimedia.org/P27371 and previous config saved to /var/cache/conftool/dbconfig/20220504-015044-ladsgroup.json [production]
01:34 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cp5002.eqsin.wmnet [production]
01:05 <sukhe@cumin2002> START - Cookbook sre.hosts.reboot-single for host cp5002.eqsin.wmnet [production]
01:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1113:3315 (T307525)', diff saved to https://phabricator.wikimedia.org/P27370 and previous config saved to /var/cache/conftool/dbconfig/20220504-010507-ladsgroup.json [production]
01:05 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
01:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
01:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T307525)', diff saved to https://phabricator.wikimedia.org/P27369 and previous config saved to /var/cache/conftool/dbconfig/20220504-010459-ladsgroup.json [production]
00:50 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance [production]
00:50 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance [production]
00:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27368 and previous config saved to /var/cache/conftool/dbconfig/20220504-004954-ladsgroup.json [production]
00:45 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp5002.eqsin.wmnet,service=ats-tls [production]
00:45 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp5002.eqsin.wmnet,service=varnish-fe [production]
00:45 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp5002.eqsin.wmnet,service=ats-be [production]
00:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27367 and previous config saved to /var/cache/conftool/dbconfig/20220504-003449-ladsgroup.json [production]
00:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T307525)', diff saved to https://phabricator.wikimedia.org/P27366 and previous config saved to /var/cache/conftool/dbconfig/20220504-001944-ladsgroup.json [production]
00:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1161 (T307525)', diff saved to https://phabricator.wikimedia.org/P27365 and previous config saved to /var/cache/conftool/dbconfig/20220504-001326-ladsgroup.json [production]
00:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
00:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
00:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
00:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
00:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1161 (re)pooling @ 100%: T307525', diff saved to https://phabricator.wikimedia.org/P27364 and previous config saved to /var/cache/conftool/dbconfig/20220504-001205-ladsgroup.json [production]
00:08 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
00:07 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
00:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
00:01 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
2022-05-03 §
23:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1161 (re)pooling @ 25%: T307525', diff saved to https://phabricator.wikimedia.org/P27363 and previous config saved to /var/cache/conftool/dbconfig/20220503-235701-ladsgroup.json [production]
23:56 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
23:50 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
23:50 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
23:50 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
23:50 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
23:50 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
23:49 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
23:48 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
23:48 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
23:48 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
23:48 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
23:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1161 (T307525)', diff saved to https://phabricator.wikimedia.org/P27362 and previous config saved to /var/cache/conftool/dbconfig/20220503-234451-ladsgroup.json [production]
23:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
23:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
23:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
23:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
23:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
23:28 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
23:27 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
23:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
23:26 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
23:23 <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:788869|Revert "cirrus: Move query traffic to codfw for maintenance" (T306811)]] (duration: 00m 56s) [production]
23:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]