3401-3450 of 10000 results (29ms)
2020-04-28 §
05:42 <marostegui> Restart labsdb1011 with innodb_purge_threads set to 10 - T249188 [production]
05:35 <marostegui> Deploy schema change on db1112 [production]
05:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1112 for schema change', diff saved to https://phabricator.wikimedia.org/P11054 and previous config saved to /var/cache/conftool/dbconfig/20200428-053453-marostegui.json [production]
04:59 <vgutierrez> depool and powercycle cp5012 [production]
04:37 <kart_> Updated cxserver to 2020-04-27-061703-production (T249852) [production]
04:34 <kartik@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
04:22 <kartik@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
04:18 <kartik@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . [production]
2020-04-27 §
23:25 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Update logos for tiwiki and tiwiktionary (T150618, T249451) (duration: 00m 57s) [production]
23:20 <catrope@deploy1001> Synchronized static/images/project-logos/: Update logos for tiwiki and tiwiktionary (T150618, T249451) (duration: 00m 58s) [production]
23:18 <catrope@deploy1001> Synchronized dblists/visualeditor-nondefault.dblist: Enable VisualEditor by default on srwiki (T250878) (duration: 00m 57s) [production]
23:16 <catrope@deploy1001> Synchronized wmf-config/config/srwiki.yaml: Enable VisualEditor by default on srwiki (T250878) (duration: 00m 58s) [production]
20:58 <bearND> mobileapps deploy on canary failed due to timeouts, rolled back. [production]
20:56 <bsitzmann@deploy1001> Finished deploy [mobileapps/deploy@99c350c]: Update mobileapps to 09cb7c2e (duration: 00m 52s) [production]
20:55 <hknust> holger@mwmaint1002 END (enwiki=success, frwiki=fail) uppercaseTitlesForUnicodeTransition.php as part of T219279 [production]
20:55 <bsitzmann@deploy1001> Started deploy [mobileapps/deploy@99c350c]: Update mobileapps to 09cb7c2e [production]
20:43 <bearND> mobileapps deployed failed due to timeouts, rolled back. [production]
20:42 <bsitzmann@deploy1001> Finished deploy [mobileapps/deploy@99c350c]: Update mobileapps to 09cb7c2e (duration: 06m 24s) [production]
20:35 <bsitzmann@deploy1001> Started deploy [mobileapps/deploy@99c350c]: Update mobileapps to 09cb7c2e [production]
20:28 <hknust> holger@mwmaint1002 Restarting uppercaseTitlesForUnicodeTransition.php as part of T219279 for 2 wikis [production]
19:26 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:25 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:23 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@ecca66b]: Switch off rules moved to k8s T248677 (duration: 01m 22s) [production]
19:22 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:21 <ppchelko@deploy1001> Started deploy [changeprop/deploy@ecca66b]: Switch off rules moved to k8s T248677 [production]
19:21 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:20 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:20 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:06 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:05 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:05 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:04 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
18:50 <James_F> Manually ran `scap pull` on mw1279.eqiad.wmnet as it flaked during deploy. [production]
18:48 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Ready wmgVisualEditorAllowExternalLinkPaste to set wgVisualEditorAllowExternalLinkPaste (duration: 01m 29s) [production]
18:48 <James_F> Sync failure to mw1279.eqiad.wmnet (timeout) [production]
18:46 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: IS: Set wmgVisualEditorAllowExternalLinkPaste false everywhere except officewiki (duration: 01m 17s) [production]
18:25 <Urbanecm> Run namespaceDupes.php for thwikisource (T251134) [production]
18:24 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 56a447e: Create several namespace aliases for thwikisource (T251134) (duration: 00m 58s) [production]
18:21 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.28/extensions/Kartographer/modules/: SWAT: 6cd2847: Do not use remove() on maplinks (T250620; T251053) (duration: 00m 58s) [production]
18:10 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 8b71f38: Remove use of `wgAllowImageMoving` (T245293) (duration: 00m 57s) [production]
18:04 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: wgEventStreams: in beta, merge settings from production - T242122 (duration: 00m 56s) [production]
18:02 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: wgEventStreams: configure SearchSatisfaction - T249261 (duration: 00m 58s) [production]
17:34 <ppchelko@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
17:31 <ppchelko@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
16:46 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
16:07 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
16:02 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
15:54 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:45 <gehel> restart wdqs-updater on all servers [production]
15:39 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]