|
2023-01-05
ยง
|
| 15:14 |
<cgoubert@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97) |
[production] |
| 15:10 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 15:10 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 15:10 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 15:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T326156)', diff saved to https://phabricator.wikimedia.org/P42868 and previous config saved to /var/cache/conftool/dbconfig/20230105-150939-ladsgroup.json |
[production] |
| 15:09 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 15:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1113:3315 (T326156)', diff saved to https://phabricator.wikimedia.org/P42867 and previous config saved to /var/cache/conftool/dbconfig/20230105-150825-ladsgroup.json |
[production] |
| 15:08 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:08 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110 (T326156)', diff saved to https://phabricator.wikimedia.org/P42866 and previous config saved to /var/cache/conftool/dbconfig/20230105-150804-ladsgroup.json |
[production] |
| 14:58 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-cluster |
[production] |
| 14:58 |
<cgoubert@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97) |
[production] |
| 14:56 |
<claime> |
hard resetting mw1486 |
[production] |
| 14:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P42865 and previous config saved to /var/cache/conftool/dbconfig/20230105-145257-ladsgroup.json |
[production] |
| 14:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P42864 and previous config saved to /var/cache/conftool/dbconfig/20230105-143751-ladsgroup.json |
[production] |
| 14:30 |
<mlitn@deploy1002> |
Finished scap: Backport for [[gerrit:875908|Also get central description (T325831)]] (duration: 08m 32s) |
[production] |
| 14:23 |
<mlitn@deploy1002> |
mlitn and mlitn: Backport for [[gerrit:875908|Also get central description (T325831)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 14:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110 (T326156)', diff saved to https://phabricator.wikimedia.org/P42862 and previous config saved to /var/cache/conftool/dbconfig/20230105-142244-ladsgroup.json |
[production] |
| 14:21 |
<mlitn@deploy1002> |
Started scap: Backport for [[gerrit:875908|Also get central description (T325831)]] |
[production] |
| 14:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1110 (T326156)', diff saved to https://phabricator.wikimedia.org/P42861 and previous config saved to /var/cache/conftool/dbconfig/20230105-142029-ladsgroup.json |
[production] |
| 14:20 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1110.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:20 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1110.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1100 (T326156)', diff saved to https://phabricator.wikimedia.org/P42860 and previous config saved to /var/cache/conftool/dbconfig/20230105-142008-ladsgroup.json |
[production] |
| 14:17 |
<mlitn@deploy1002> |
Finished scap: Backport for [[gerrit:875906|Also get central description (T325831)]] (duration: 07m 57s) |
[production] |
| 14:11 |
<mlitn@deploy1002> |
mlitn and mlitn: Backport for [[gerrit:875906|Also get central description (T325831)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 14:09 |
<mlitn@deploy1002> |
Started scap: Backport for [[gerrit:875906|Also get central description (T325831)]] |
[production] |
| 14:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P42859 and previous config saved to /var/cache/conftool/dbconfig/20230105-140501-ladsgroup.json |
[production] |
| 13:58 |
<Amir1> |
start of externallinks migration in elwiki (and rest of large wikis in s3) (T326314) |
[production] |
| 13:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P42858 and previous config saved to /var/cache/conftool/dbconfig/20230105-134955-ladsgroup.json |
[production] |
| 13:46 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:875892|Enable write both for externallinks in ten largest s3 wikis (T321662)]] (duration: 08m 54s) |
[production] |
| 13:42 |
<urbanecm> |
aswikiquote: Run importDump.php to import a XML dump (per new wiki importers request, running into issues with a largish page) |
[production] |
| 13:39 |
<ladsgroup@deploy1002> |
ladsgroup and ladsgroup: Backport for [[gerrit:875892|Enable write both for externallinks in ten largest s3 wikis (T321662)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 13:38 |
<XioNoX> |
start [eqiad] faulty VC optics maintenance - T325803 |
[production] |
| 13:37 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:875892|Enable write both for externallinks in ten largest s3 wikis (T321662)]] |
[production] |
| 13:34 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1100 (T326156)', diff saved to https://phabricator.wikimedia.org/P42857 and previous config saved to /var/cache/conftool/dbconfig/20230105-133448-ladsgroup.json |
[production] |
| 13:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1100 (T326156)', diff saved to https://phabricator.wikimedia.org/P42856 and previous config saved to /var/cache/conftool/dbconfig/20230105-133234-ladsgroup.json |
[production] |
| 13:32 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1100.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:32 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1100.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T326156)', diff saved to https://phabricator.wikimedia.org/P42855 and previous config saved to /var/cache/conftool/dbconfig/20230105-133211-ladsgroup.json |
[production] |
| 13:30 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 13:29 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 13:21 |
<effie> |
enable puppet on all mw servers |
[production] |
| 13:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P42854 and previous config saved to /var/cache/conftool/dbconfig/20230105-131705-ladsgroup.json |
[production] |
| 13:03 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 13:03 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 13:03 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 13:03 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 13:03 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply |
[production] |
| 13:02 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-api-int: apply |
[production] |
| 13:02 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply |
[production] |