|
2020-07-13
ยง
|
| 19:41 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@de0a1f1]: Regular analytics weekly train [analytics/refinery@de0a1f1] |
[production] |
| 19:39 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:33 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: I1a12124f1811e9a (duration: 00m 57s) |
[production] |
| 18:53 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T248343 Don't use the 'zeroconf' configuration for VisualEditor (duration: 00m 55s) |
[production] |
| 18:43 |
<dcausse> |
BACON done |
[production] |
| 18:40 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T257745: Add rollbacker to elwiki (duration: 00m 56s) |
[production] |
| 18:26 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T250810: Set proper language code for some wikis (duration: 00m 56s) |
[production] |
| 18:18 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T256928: Scale largest shards to be closer to 30GB (duration: 00m 56s) |
[production] |
| 16:17 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:17 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:56 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/Wikibase.php: [[gerrit:610265|Load WikibaseClient using extension registration in beta (T257435)]] (duration: 00m 55s) |
[production] |
| 15:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1130', diff saved to https://phabricator.wikimedia.org/P11882 and previous config saved to /var/cache/conftool/dbconfig/20200713-155240-marostegui.json |
[production] |
| 15:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11881 and previous config saved to /var/cache/conftool/dbconfig/20200713-154847-marostegui.json |
[production] |
| 15:39 |
<mholloway-shell@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 15:35 |
<mholloway-shell@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 15:30 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 14:50 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Stop setting DiscussionToolsEnableVisual, default value (duration: 00m 57s) |
[production] |
| 14:17 |
<moritzm> |
removing lilypond from production T257066 |
[production] |
| 13:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11880 and previous config saved to /var/cache/conftool/dbconfig/20200713-133604-marostegui.json |
[production] |
| 13:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1082', diff saved to https://phabricator.wikimedia.org/P11879 and previous config saved to /var/cache/conftool/dbconfig/20200713-133535-marostegui.json |
[production] |
| 13:05 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1022, and set es1020 to zero weight T257284', diff saved to https://phabricator.wikimedia.org/P11878 and previous config saved to /var/cache/conftool/dbconfig/20200713-130532-kormat.json |
[production] |
| 12:08 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Start repooling es1022 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11873 and previous config saved to /var/cache/conftool/dbconfig/20200713-120818-kormat.json |
[production] |
| 11:49 |
<Urbanecm> |
Password reset for User:Alert5 (T257806) |
[production] |
| 11:44 |
<akosiaris> |
repool ganeti1007 T244530. Start emptying ganeti1008 |
[production] |
| 11:08 |
<Urbanecm> |
EU B&C done |
[production] |
| 11:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 896c042296b4e1f5d88f786981537655e5d9fea9: Enable SandboxLink extension in trwiki (T256782) (duration: 00m 56s) |
[production] |
| 10:44 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:612175| Bumping portals to master (612175)]] (duration: 00m 56s) |
[production] |
| 10:43 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:612175| Bumping portals to master (612175)]] (duration: 00m 56s) |
[production] |
| 09:42 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:39 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:58 |
<ema> |
cp: rolling ats-backend-restart to apply SyslogIdentifier changes -> https://gerrit.wikimedia.org/r/c/operations/puppet/+/611311 |
[production] |
| 08:57 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T248343 Explicitly set visualeditor-enable to 0 when non-default (duration: 00m 57s) |
[production] |
| 08:44 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Depool es1022 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11871 and previous config saved to /var/cache/conftool/dbconfig/20200713-084449-kormat.json |
[production] |
| 08:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1093', diff saved to https://phabricator.wikimedia.org/P11870 and previous config saved to /var/cache/conftool/dbconfig/20200713-083902-marostegui.json |
[production] |
| 08:34 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1022 T257284', diff saved to https://phabricator.wikimedia.org/P11869 and previous config saved to /var/cache/conftool/dbconfig/20200713-083414-kormat.json |
[production] |
| 08:20 |
<kormat> |
reimaging es1022 T257284 |
[production] |
| 06:54 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 06:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:52 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 06:52 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:51 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 06:50 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:16 |
<marostegui> |
Reverse gerrit password on m2 master - T255715 |
[production] |
| 06:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11868 and previous config saved to /var/cache/conftool/dbconfig/20200713-060410-marostegui.json |
[production] |
| 05:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11867 and previous config saved to /var/cache/conftool/dbconfig/20200713-055422-marostegui.json |
[production] |
| 05:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093 for upgrade', diff saved to https://phabricator.wikimedia.org/P11866 and previous config saved to /var/cache/conftool/dbconfig/20200713-054840-marostegui.json |
[production] |
| 05:34 |
<marostegui> |
Deploy schema change on s3 codfw master, lag will appear on codfw T253276 |
[production] |
| 05:30 |
<marostegui> |
Stop replication on db1082 for schema change and triggers removal T238966 |
[production] |
| 05:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P11865 and previous config saved to /var/cache/conftool/dbconfig/20200713-052928-marostegui.json |
[production] |
| 05:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1119 for innodb compression', diff saved to https://phabricator.wikimedia.org/P11864 and previous config saved to /var/cache/conftool/dbconfig/20200713-051428-marostegui.json |
[production] |