1251-1300 of 10000 results (23ms)
2020-05-05 §
06:59 <addshore> depool wdqs1006 heavy lag [production]
05:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s5 and s6 as read-only=off for maintenance T251154', diff saved to https://phabricator.wikimedia.org/P11133 and previous config saved to /var/cache/conftool/dbconfig/20200505-052334-marostegui.json [production]
05:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s5 and s6 as read-only for maintenance T251154', diff saved to https://phabricator.wikimedia.org/P11132 and previous config saved to /var/cache/conftool/dbconfig/20200505-052058-marostegui.json [production]
05:19 <marostegui> Start s5 and s6 maintenance - T251154 [production]
04:39 <marostegui> Restart mysql on tendril host: db1115 - T231769 [production]
2020-05-04 §
23:38 <mstyles@deploy1001> Finished deploy [wdqs/wdqs@6518a8d]: v.0.3.26 (duration: 14m 39s) [production]
23:37 <reedy@deploy1001> Synchronized wmf-config/CommonSettings.php: Use namespaced EventBus classes (duration: 00m 57s) [production]
23:35 <reedy@deploy1001> Synchronized wmf-config/logging.php: Use namespaced EventBus classes (duration: 00m 56s) [production]
23:33 <reedy@deploy1001> Synchronized rpc/RunSingleJob.php: Use namespaced EventBus classes (duration: 00m 58s) [production]
23:29 <reedy@deploy1001> Synchronized wmf-config/logging.php: Replace AuthManagerStatsdHandler with WikimediaEventsAuthManagerStatsdHandler::class (duration: 00m 57s) [production]
23:23 <mstyles@deploy1001> Started deploy [wdqs/wdqs@6518a8d]: v.0.3.26 [production]
22:42 <sbassett@deploy1001> Synchronized private/PrivateSettings.php: T251835: Restore dc752af1e94684faacbe9662789815c6edbbdf46 (duration: 00m 57s) [production]
22:16 <eileen> process-control config revision is 2eb75f8dff [production]
22:06 <sbassett@deploy1001> Synchronized private/PrivateSettings.php: Partial mitigation for T250887 (duration: 00m 57s) [production]
21:45 <sbassett@deploy1001> Synchronized private/PrivateSettings.php: Revert partial mitigation for T250887 (duration: 00m 57s) [production]
21:41 <sbassett@deploy1001> Synchronized private/PrivateSettings.php: Deploy partial mitigation for T250887 (duration: 00m 57s) [production]
18:20 <dpifke@deploy1001> Finished deploy [performance/navtiming@239d359]: Deploy navtiming with new/updated Prometheus metrics - T249822, T238086 (duration: 00m 05s) [production]
18:19 <dpifke@deploy1001> Started deploy [performance/navtiming@239d359]: Deploy navtiming with new/updated Prometheus metrics - T249822, T238086 [production]
18:16 <Urbanecm> Morning SWAT done [production]
18:15 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: c04fbdd: Adding upload_by_url user right to all registered users on Commons (T251474) (duration: 00m 57s) [production]
18:11 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.30/extensions/DiscussionTools/includes/DiscussionToolsHooks.php: SWAT: b85fc16: Enable on all ExtraSignaturesNamespaces (T249036) (duration: 01m 00s) [production]
18:07 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 18c1efb: Load DiscussionTools on en.wiki (T249376) (duration: 00m 58s) [production]
17:57 <XioNoX> configure singtel interface on cr1-eqsin [production]
17:36 <volans> upgraded spicerack on cumin[12]001 to 0.0.33-1 [production]
17:02 <joal@deploy1001> Finished deploy [analytics/refinery@2252f9a] (thin): Analytics hotfix deploy 2 THIN (sqoop) [2252f9a] (duration: 00m 09s) [production]
17:02 <joal@deploy1001> Started deploy [analytics/refinery@2252f9a] (thin): Analytics hotfix deploy 2 THIN (sqoop) [2252f9a] [production]
17:01 <joal@deploy1001> Finished deploy [analytics/refinery@2252f9a]: Analytics hotfix deploy 2 (sqoop) [2252f9a] (duration: 16m 45s) [production]
16:44 <joal@deploy1001> Started deploy [analytics/refinery@2252f9a]: Analytics hotfix deploy 2 (sqoop) [2252f9a] [production]
16:08 <liw@deploy1001> rebuilt and synchronized wikiversions files: group2 wikis to 1.35.0-wmf.30 [production]
15:59 <liw@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.30 (duration: 01m 05s) [production]
15:58 <liw@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.30 [production]
15:53 <root@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
15:53 <root@cumin1001> Updating IPMI password on 1 hosts - root@cumin1001 [production]
15:53 <root@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
15:52 <root@cumin1001> END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99) [production]
15:52 <root@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
15:47 <kormat@cumin1001> dbctl commit (dc=all): 'Repool es2025 after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11128 and previous config saved to /var/cache/conftool/dbconfig/20200504-154747-kormat.json [production]
15:45 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.30/includes/libs/rdbms/database/DatabaseMysqlBase.php: T251457 rdbms: don't treat lock() as a write operation (duration: 01m 04s) [production]
15:43 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.30/resources/src/mediawiki.diff.styles/diff.less: T250393 Follow-up I07dd6f7: Fix font size in diff (duration: 01m 05s) [production]
15:34 <volans> uploaded spicerack_0.0.33-1_amd64.deb to apt.wikimedia.org stretch-wikimedia [production]
15:26 <volans> deploy1001: deleted old .hhvm.hhbc files (/home/*/.hhvm.hhbc) https://phabricator.wikimedia.org/P11127 [production]
15:23 <volans> deploy1001: deleted old .hhvm.hhbc files moved from tin (/home/*/home-tin/.hhvm.hhbc) https://phabricator.wikimedia.org/P11126 [production]
15:12 <kormat@cumin1001> dbctl commit (dc=all): 'Repool db1101:3318 fully after reimaging T250666', diff saved to https://phabricator.wikimedia.org/P11125 and previous config saved to /var/cache/conftool/dbconfig/20200504-151243-kormat.json [production]
15:11 <ppchelko@deploy1001> Finished deploy [restbase/deploy@74db57e]: Enable greek community wiki, fix analytics endpoints (duration: 14m 36s) [production]
15:05 <joal@deploy1001> Finished deploy [analytics/refinery@3396279] (thin): Analytics hotfix deploy (sqoop) THIN [3396279] (duration: 00m 10s) [production]
15:05 <joal@deploy1001> Started deploy [analytics/refinery@3396279] (thin): Analytics hotfix deploy (sqoop) THIN [3396279] [production]
15:05 <joal@deploy1001> Finished deploy [analytics/refinery@3396279]: Analytics hotfix deploy (sqoop) [3396279] (duration: 15m 07s) [production]
15:05 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:57 <ppchelko@deploy1001> Started deploy [restbase/deploy@74db57e]: Enable greek community wiki, fix analytics endpoints [production]