9351-9400 of 10000 results (107ms)
2019-04-11 §
05:00 <marostegui> Starting s3 failover from db1078 to db1075 - T219115 [production]
04:32 <marostegui> Disable puppet on db1078 and db1075 T219115 [production]
04:18 <marostegui> Start topology changes to move s3 slaves under db1075 T219115 [production]
04:14 <marostegui> Disable GTID on s3 hosts - https://phabricator.wikimedia.org/T219115 [production]
02:33 <andrewbogott> tools-paws-worker-1005, tools-paws-worker-1006 to eqiad1-r [tools]
00:45 <jforrester@deploy1001> Synchronized php-1.33.0-wmf.25/extensions/PageTriage/: UBN Fix for pageTriage and ORES T220649 (duration: 01m 04s) [production]
00:12 <twentyafterfour> deploying phabricator upgrade [production]
00:03 <andrewbogott> tools-paws-worker-1002, tools-paws-worker-1003 to eqiad1-r [tools]
2019-04-10 §
23:58 <andrewbogott> moving tools-clushmaster-02, tools-elastic-03 and tools-paws-worker-1001 to eqiad1-r [tools]
23:32 <James_F> Manually created REL1_33 branches for the core, vendor, and tarball extensions and skins. Eurgh. T220653 [releng]
22:36 <James_F> Deleted faulty REL1_33 branches for the Timeless, Vector and Monobook skins; they were duplicates of the REL1_32 branches. [releng]
21:35 <bstorm_> shut down webservice. It never runs at this time. It appears to need venv rebuilt. [tools.mediaviews-api]
20:43 <urandom> decommissioning cassandra-c, restbase2007 -- T208087 [production]
20:27 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Revert - Enabling api-request logging via eventgate-analytics for group1 wikis - T214080 (duration: 01m 00s) [production]
20:27 <paladox> create operations/software/gerrit/plugins/MassBranchCreation repository [releng]
19:48 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enabling api-request logging via eventgate-analytics for group1 wikis - T214080 (duration: 00m 59s) [production]
19:42 <twentyafterfour@deploy1001> Synchronized php: group1 wikis to 1.33.0-wmf.25 refs T206679 (duration: 01m 48s) [production]
19:40 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.25 refs T206679 [production]
19:28 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.33.0-wmf.25 refs T206679 [production]
19:26 <XioNoX> enable sampling on cr2-eqiad external links, outbound [production]
19:23 <bstorm_> T220646 shut down webservice because it is just in a restart loop and cannot run [tools.dplbot]
19:17 <twentyafterfour@deploy1001> Pruned MediaWiki: 1.33.0-wmf.20 [keeping static files] (duration: 02m 18s) [production]
19:14 <ejegg> updated fundraising CiviCRM from d0e44a9e51 to 24b968b1f9 [production]
19:08 <twentyafterfour@deploy1001> Pruned MediaWiki: 1.33.0-wmf.19 [keeping static files] (duration: 02m 22s) [production]
18:52 <bstorm_> depooled and rebooted tools-sgeexec-0929 because systemd was in a weird state [tools]
18:46 <bstorm_> depooled and rebooted tools-sgewebgrid-lighttpd-0913 because high load was caused by ancient lsof processes [tools]
17:44 <twentyafterfour@deploy1001> Pruned MediaWiki: 1.33.0-wmf.18 [keeping static files] (duration: 02m 22s) [production]
16:58 <chaomodus> restarted nagios-nrpe-server on proton1001 (it died due to OOM) [production]
16:51 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=druid1004.eqiad.wmnet [production]
16:01 <elukey> restart brokers on druid100[3-6] - locking after segments get deleted [production]
15:46 <krinkle@deploy1001> Synchronized php-1.33.0-wmf.25/includes/parser/DateFormatter.php: Ib2b3fb315dc93b / T220563 (duration: 01m 00s) [production]
15:28 <gilles@deploy1001> Synchronized php-1.33.0-wmf.25/includes/media/ThumbnailImage.php: T216499 Only apply high priority hint half the time (duration: 00m 59s) [production]
15:26 <oblivian@deploy1001> Finished deploy [docker-pkg/deploy@605690c]: Upgrade to docker-pkg 2.0.0 everywhere (duration: 00m 21s) [production]
15:26 <oblivian@deploy1001> Started deploy [docker-pkg/deploy@605690c]: Upgrade to docker-pkg 2.0.0 everywhere [production]
15:24 <jforrester@deploy1001> Synchronized php-1.33.0-wmf.25/extensions/Score/: UBN Revert Score changes that broke VE T220465 (duration: 01m 01s) [production]
15:19 <oblivian@deploy1001> Finished deploy [docker-pkg/deploy@605690c]: Upgrade to docker-pkg 2.0.0 (duration: 00m 13s) [production]
15:19 <oblivian@deploy1001> Started deploy [docker-pkg/deploy@605690c]: Upgrade to docker-pkg 2.0.0 [production]
15:01 <fsero> pooled back mwdebug200[1,2] T219989 [production]
15:00 <fsero> repooling mwdebug2002 [production]
15:00 <jijiki> Enable puppet on thumbor1001, switch back to nginx, pool thumbor1004 - T187765 [production]
14:57 <fsero> repooling mwdebug2001 [production]
14:49 <bstorm_> cleared E state from 5 queues [tools]
14:48 <elukey> restart turnilo to pick up the new nodejs runtime [analytics]
14:20 <hashar> CI processing was a bit slower than usual over the past couple hours or so. It should be slightly faster now T220606 [production]
14:16 <hashar> contint1001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin # T220606 [releng]
14:13 <joal@deploy1001> Finished deploy [analytics/aqs/deploy@fc1d232]: Deploying per-page limits for druid-endpoints (duration: 14m 41s) [production]
14:13 <hashar> contint2001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin # T220606 [releng]
13:58 <joal> Deploying AQS [analytics]
13:58 <joal@deploy1001> Started deploy [analytics/aqs/deploy@fc1d232]: Deploying per-page limits for druid-endpoints [production]
13:47 <fsero> resizing disk on mwdebug2002 T219989 [production]