2151-2200 of 10000 results (38ms)
2020-07-13 ยง
19:48 <milimetric@deploy1001> Finished deploy [analytics/refinery@de0a1f1] (thin): Regular analytics weekly train THIN [analytics/refinery@de0a1f1] (duration: 00m 07s) [production]
19:47 <milimetric@deploy1001> Started deploy [analytics/refinery@de0a1f1] (thin): Regular analytics weekly train THIN [analytics/refinery@de0a1f1] [production]
19:47 <milimetric@deploy1001> Finished deploy [analytics/refinery@de0a1f1]: Regular analytics weekly train [analytics/refinery@de0a1f1] (duration: 06m 41s) [production]
19:41 <milimetric@deploy1001> Started deploy [analytics/refinery@de0a1f1]: Regular analytics weekly train [analytics/refinery@de0a1f1] [production]
19:39 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:33 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: I1a12124f1811e9a (duration: 00m 57s) [production]
18:53 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T248343 Don't use the 'zeroconf' configuration for VisualEditor (duration: 00m 55s) [production]
18:43 <dcausse> BACON done [production]
18:40 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T257745: Add rollbacker to elwiki (duration: 00m 56s) [production]
18:26 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T250810: Set proper language code for some wikis (duration: 00m 56s) [production]
18:18 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T256928: Scale largest shards to be closer to 30GB (duration: 00m 56s) [production]
16:17 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:17 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:56 <ladsgroup@deploy1001> Synchronized wmf-config/Wikibase.php: [[gerrit:610265|Load WikibaseClient using extension registration in beta (T257435)]] (duration: 00m 55s) [production]
15:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1130', diff saved to https://phabricator.wikimedia.org/P11882 and previous config saved to /var/cache/conftool/dbconfig/20200713-155240-marostegui.json [production]
15:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11881 and previous config saved to /var/cache/conftool/dbconfig/20200713-154847-marostegui.json [production]
15:39 <mholloway-shell@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
15:35 <mholloway-shell@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
15:30 <mholloway-shell@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
14:50 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Stop setting DiscussionToolsEnableVisual, default value (duration: 00m 57s) [production]
14:17 <moritzm> removing lilypond from production T257066 [production]
13:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11880 and previous config saved to /var/cache/conftool/dbconfig/20200713-133604-marostegui.json [production]
13:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1082', diff saved to https://phabricator.wikimedia.org/P11879 and previous config saved to /var/cache/conftool/dbconfig/20200713-133535-marostegui.json [production]
13:05 <kormat@cumin1001> dbctl commit (dc=all): 'Fully repool es1022, and set es1020 to zero weight T257284', diff saved to https://phabricator.wikimedia.org/P11878 and previous config saved to /var/cache/conftool/dbconfig/20200713-130532-kormat.json [production]
12:08 <kormat@cumin1001> dbctl commit (dc=all): 'Start repooling es1022 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11873 and previous config saved to /var/cache/conftool/dbconfig/20200713-120818-kormat.json [production]
11:49 <Urbanecm> Password reset for User:Alert5 (T257806) [production]
11:44 <akosiaris> repool ganeti1007 T244530. Start emptying ganeti1008 [production]
11:08 <Urbanecm> EU B&C done [production]
11:06 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 896c042296b4e1f5d88f786981537655e5d9fea9: Enable SandboxLink extension in trwiki (T256782) (duration: 00m 56s) [production]
10:44 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:612175| Bumping portals to master (612175)]] (duration: 00m 56s) [production]
10:43 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:612175| Bumping portals to master (612175)]] (duration: 00m 56s) [production]
09:42 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:39 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:58 <ema> cp: rolling ats-backend-restart to apply SyslogIdentifier changes -> https://gerrit.wikimedia.org/r/c/operations/puppet/+/611311 [production]
08:57 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T248343 Explicitly set visualeditor-enable to 0 when non-default (duration: 00m 57s) [production]
08:44 <kormat@cumin1001> dbctl commit (dc=all): 'Depool es1022 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11871 and previous config saved to /var/cache/conftool/dbconfig/20200713-084449-kormat.json [production]
08:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1093', diff saved to https://phabricator.wikimedia.org/P11870 and previous config saved to /var/cache/conftool/dbconfig/20200713-083902-marostegui.json [production]
08:34 <kormat@cumin1001> dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1022 T257284', diff saved to https://phabricator.wikimedia.org/P11869 and previous config saved to /var/cache/conftool/dbconfig/20200713-083414-kormat.json [production]
08:20 <kormat> reimaging es1022 T257284 [production]
06:54 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
06:53 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:52 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
06:52 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:51 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) [production]
06:50 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:16 <marostegui> Reverse gerrit password on m2 master - T255715 [production]
06:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11868 and previous config saved to /var/cache/conftool/dbconfig/20200713-060410-marostegui.json [production]
05:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11867 and previous config saved to /var/cache/conftool/dbconfig/20200713-055422-marostegui.json [production]
05:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1093 for upgrade', diff saved to https://phabricator.wikimedia.org/P11866 and previous config saved to /var/cache/conftool/dbconfig/20200713-054840-marostegui.json [production]
05:34 <marostegui> Deploy schema change on s3 codfw master, lag will appear on codfw T253276 [production]