1201-1250 of 10000 results (50ms)
2022-03-29 §
06:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23456 and previous config saved to /var/cache/conftool/dbconfig/20220329-064013-ladsgroup.json [production]
06:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298556)', diff saved to https://phabricator.wikimedia.org/P23455 and previous config saved to /var/cache/conftool/dbconfig/20220329-062912-marostegui.json [production]
06:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181 (T298565)', diff saved to https://phabricator.wikimedia.org/P23454 and previous config saved to /var/cache/conftool/dbconfig/20220329-062625-ladsgroup.json [production]
06:25 <marostegui> dbmaint s3@eqiad T300775 [production]
06:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P23453 and previous config saved to /var/cache/conftool/dbconfig/20220329-062508-ladsgroup.json [production]
06:17 <marostegui> dbmaint s3@eqiad T300381 [production]
06:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P23452 and previous config saved to /var/cache/conftool/dbconfig/20220329-061407-marostegui.json [production]
06:11 <marostegui> Maintenance on db1157 (old s3 master) T301848 [production]
06:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P23451 and previous config saved to /var/cache/conftool/dbconfig/20220329-061004-ladsgroup.json [production]
06:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1157 T301850', diff saved to https://phabricator.wikimedia.org/P23450 and previous config saved to /var/cache/conftool/dbconfig/20220329-060532-root.json [production]
06:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1123 to s3 primary and set section read-write T301850', diff saved to https://phabricator.wikimedia.org/P23449 and previous config saved to /var/cache/conftool/dbconfig/20220329-060059-marostegui.json [production]
06:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s3 eqiad as read-only for maintenance - T301850', diff saved to https://phabricator.wikimedia.org/P23448 and previous config saved to /var/cache/conftool/dbconfig/20220329-060024-marostegui.json [production]
06:00 <marostegui> Starting s3 eqiad failover from db1157 to db1123 - T301850 [production]
05:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P23447 and previous config saved to /var/cache/conftool/dbconfig/20220329-055902-marostegui.json [production]
05:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1181 (T298565)', diff saved to https://phabricator.wikimedia.org/P23446 and previous config saved to /var/cache/conftool/dbconfig/20220329-055544-ladsgroup.json [production]
05:55 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
05:55 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
05:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23445 and previous config saved to /var/cache/conftool/dbconfig/20220329-055458-ladsgroup.json [production]
05:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23444 and previous config saved to /var/cache/conftool/dbconfig/20220329-055251-ladsgroup.json [production]
05:52 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
05:52 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
05:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298556)', diff saved to https://phabricator.wikimedia.org/P23443 and previous config saved to /var/cache/conftool/dbconfig/20220329-054357-marostegui.json [production]
05:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1148 (T298556)', diff saved to https://phabricator.wikimedia.org/P23442 and previous config saved to /var/cache/conftool/dbconfig/20220329-052331-marostegui.json [production]
05:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance [production]
05:23 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance [production]
05:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1134 (T300775)', diff saved to https://phabricator.wikimedia.org/P23441 and previous config saved to /var/cache/conftool/dbconfig/20220329-051951-marostegui.json [production]
05:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
05:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
05:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T300775)', diff saved to https://phabricator.wikimedia.org/P23440 and previous config saved to /var/cache/conftool/dbconfig/20220329-051943-marostegui.json [production]
05:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P23439 and previous config saved to /var/cache/conftool/dbconfig/20220329-050438-marostegui.json [production]
05:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1123 with weight 0 T301850', diff saved to https://phabricator.wikimedia.org/P23438 and previous config saved to /var/cache/conftool/dbconfig/20220329-050234-root.json [production]
05:02 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Primary switchover s3 T301850 [production]
05:02 <root@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 20 hosts with reason: Primary switchover s3 T301850 [production]
04:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P23437 and previous config saved to /var/cache/conftool/dbconfig/20220329-044933-marostegui.json [production]
04:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T300775)', diff saved to https://phabricator.wikimedia.org/P23436 and previous config saved to /var/cache/conftool/dbconfig/20220329-043428-marostegui.json [production]
02:30 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:30 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
02:08 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
2022-03-28 §
23:15 <eileen> civicrm revision 15d22bd1 -> 1c5d10e1 [production]
23:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1184 (T300775)', diff saved to https://phabricator.wikimedia.org/P23434 and previous config saved to /var/cache/conftool/dbconfig/20220328-230012-marostegui.json [production]
23:00 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance [production]
23:00 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance [production]
23:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1164 (T300775)', diff saved to https://phabricator.wikimedia.org/P23433 and previous config saved to /var/cache/conftool/dbconfig/20220328-230004-marostegui.json [production]
22:52 <ejegg> updated fundraising python tools from 409c80b7 to 8f5119f6 [production]
22:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P23431 and previous config saved to /var/cache/conftool/dbconfig/20220328-224459-marostegui.json [production]