|
2022-03-29
ยง
|
| 07:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P23472 and previous config saved to /var/cache/conftool/dbconfig/20220329-075611-ladsgroup.json |
[production] |
| 07:48 |
<marostegui> |
dbmaint s3@eqiad T298554 |
[production] |
| 07:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P23471 and previous config saved to /var/cache/conftool/dbconfig/20220329-074509-ladsgroup.json |
[production] |
| 07:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P23470 and previous config saved to /var/cache/conftool/dbconfig/20220329-074106-ladsgroup.json |
[production] |
| 07:37 |
<marostegui> |
dbmaint s6@eqiad T297189 |
[production] |
| 07:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1096:3316 for schema change', diff saved to https://phabricator.wikimedia.org/P23469 and previous config saved to /var/cache/conftool/dbconfig/20220329-073703-root.json |
[production] |
| 07:36 |
<ayounsi@cumin1001> |
START - Cookbook sre.hosts.reimage for host hppxetest2001.codfw.wmnet with OS bullseye |
[production] |
| 07:35 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host hppxetest2001.codfw.wmnet |
[production] |
| 07:34 |
<ayounsi@cumin1001> |
START - Cookbook sre.hosts.dhcp for host hppxetest2001.codfw.wmnet |
[production] |
| 07:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23468 and previous config saved to /var/cache/conftool/dbconfig/20220329-073004-ladsgroup.json |
[production] |
| 07:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23467 and previous config saved to /var/cache/conftool/dbconfig/20220329-072756-ladsgroup.json |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:27 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:27 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23466 and previous config saved to /var/cache/conftool/dbconfig/20220329-072744-ladsgroup.json |
[production] |
| 07:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 07:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 07:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P23465 and previous config saved to /var/cache/conftool/dbconfig/20220329-072601-ladsgroup.json |
[production] |
| 07:24 |
<taavi> |
UTC morning deploys done |
[production] |
| 07:23 |
<kartik@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:774386|Add viwiki eliminators to wgContentTranslationPublishRequirements (T299636)]] (duration: 00m 50s) |
[production] |
| 07:23 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 07:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23464 and previous config saved to /var/cache/conftool/dbconfig/20220329-071239-ladsgroup.json |
[production] |
| 07:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1174 (T298565)', diff saved to https://phabricator.wikimedia.org/P23463 and previous config saved to /var/cache/conftool/dbconfig/20220329-071148-ladsgroup.json |
[production] |
| 07:11 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:11 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T298565)', diff saved to https://phabricator.wikimedia.org/P23462 and previous config saved to /var/cache/conftool/dbconfig/20220329-071140-ladsgroup.json |
[production] |
| 06:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23461 and previous config saved to /var/cache/conftool/dbconfig/20220329-065734-ladsgroup.json |
[production] |
| 06:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P23460 and previous config saved to /var/cache/conftool/dbconfig/20220329-065635-ladsgroup.json |
[production] |
| 06:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23459 and previous config saved to /var/cache/conftool/dbconfig/20220329-064229-ladsgroup.json |
[production] |
| 06:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P23458 and previous config saved to /var/cache/conftool/dbconfig/20220329-064130-ladsgroup.json |
[production] |
| 06:40 |
<_joe_> |
restarting varnish text-fe on cp1079 |
[production] |
| 06:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23457 and previous config saved to /var/cache/conftool/dbconfig/20220329-064021-ladsgroup.json |
[production] |
| 06:40 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:40 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23456 and previous config saved to /var/cache/conftool/dbconfig/20220329-064013-ladsgroup.json |
[production] |
| 06:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298556)', diff saved to https://phabricator.wikimedia.org/P23455 and previous config saved to /var/cache/conftool/dbconfig/20220329-062912-marostegui.json |
[production] |
| 06:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T298565)', diff saved to https://phabricator.wikimedia.org/P23454 and previous config saved to /var/cache/conftool/dbconfig/20220329-062625-ladsgroup.json |
[production] |
| 06:25 |
<marostegui> |
dbmaint s3@eqiad T300775 |
[production] |
| 06:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P23453 and previous config saved to /var/cache/conftool/dbconfig/20220329-062508-ladsgroup.json |
[production] |
| 06:17 |
<marostegui> |
dbmaint s3@eqiad T300381 |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P23452 and previous config saved to /var/cache/conftool/dbconfig/20220329-061407-marostegui.json |
[production] |
| 06:11 |
<marostegui> |
Maintenance on db1157 (old s3 master) T301848 |
[production] |
| 06:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P23451 and previous config saved to /var/cache/conftool/dbconfig/20220329-061004-ladsgroup.json |
[production] |
| 06:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1157 T301850', diff saved to https://phabricator.wikimedia.org/P23450 and previous config saved to /var/cache/conftool/dbconfig/20220329-060532-root.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1123 to s3 primary and set section read-write T301850', diff saved to https://phabricator.wikimedia.org/P23449 and previous config saved to /var/cache/conftool/dbconfig/20220329-060059-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s3 eqiad as read-only for maintenance - T301850', diff saved to https://phabricator.wikimedia.org/P23448 and previous config saved to /var/cache/conftool/dbconfig/20220329-060024-marostegui.json |
[production] |
| 06:00 |
<marostegui> |
Starting s3 eqiad failover from db1157 to db1123 - T301850 |
[production] |
| 05:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P23447 and previous config saved to /var/cache/conftool/dbconfig/20220329-055902-marostegui.json |
[production] |