6851-6900 of 10000 results (50ms)
2016-02-18 §
01:51 <twentyafterfour> phab pre-upgrade: http://pastebin.com/RTmXfDhp [production]
01:49 <twentyafterfour> about to bring down phabricator to do the upgrade [production]
01:49 <twentyafterfour> ran puppet on iridium for testing [production]
01:08 <twentyafterfour> stopped phd and started dumping phabricator's database to /srv/dumps/20160218.phabricator.sql.gz (just in case I need to roll back the update) [production]
00:34 <catrope@tin> Synchronized php-1.27.0-wmf.13/extensions/Flow: Trying again (duration: 01m 50s) [production]
00:28 <RoanKattouw> 00:28:25 64 apaches had sync errors , /usr/bin/sync-common missing [production]
00:28 <catrope@tin> Synchronized php-1.27.0-wmf.13/extensions/Flow: SWAT (duration: 02m 06s) [production]
00:18 <godog> restart cassandra-a on restbase1008 after extending /srv [production]
2016-02-17 §
23:53 <csteipp> redeployed wmf14 patches [production]
23:30 <csteipp> deployed all missing security patches from wmf14 [production]
23:10 <csteipp@tin> Synchronized php-1.27.0-wmf.14/resources/src/mediawiki/page/patrol.ajax.js: add security patches (duration: 01m 28s) [production]
23:08 <csteipp@tin> Synchronized php-1.27.0-wmf.14/includes: add security patches (duration: 01m 35s) [production]
23:03 <ori@mira> Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: live-hacked debug logging for T124356 (duration: 02m 16s) [production]
21:42 <mobrovac> mathoid deploying ed98ffe9d [production]
21:35 <mobrovac> restbase restarted restbase1002 on nodejs v4.3.0 [production]
20:40 <papaul> es201[1-9] - signing puppet certs, salt-key, initial run [production]
20:25 <krinkle@tin> Synchronized wmf-config/CommonSettings.php: Re-enable T99096 for mediawiki.org (duration: 01m 29s) [production]
20:23 <catrope@tin> Synchronized docroot/: (no message) (duration: 01m 33s) [production]
19:18 <yuvipanda> truncate 1.2T php error log file on labstore1003 from cluebot [production]
18:35 <jynus> testing now that alerts still work by stopping db1024 replication (depooled) [production]
18:30 <krinkle@tin> Synchronized wmf-config/CommonSettings.php: T127194 (duration: 01m 31s) [production]
18:27 <jynus> no issues found with new mysql, lag monitoring, renabling puppet again on the pending eqiad servers [production]
17:49 <bblack> restarting pybal on eqiad primary LVS ( lvs100[123] ) [production]
17:47 <bblack> restarting pybal on codfw primary LVS ( lvs200[123]) [production]
17:42 <bblack> restarting pybal on ulsfo/esams primary LVS ( lvs[34]00[12]) [production]
17:40 <bblack> restarting pybal on eqiad backup LVS ( lvs100[456] ) [production]
17:38 <bblack> restarting pybal on eqiad inactive LVS clusters ( lvs1007-12 ) [production]
17:38 <bblack> restarting pybal on codfw backup LVS ( lvs200[456] ) [production]
17:34 <bblack> restarting pybal on ulsfo/esams backup LVS ( lvs[34]00[34]) [production]
17:13 <hoo> Updated the sites and site_identifiers table for on all non-Wikipedias (including Wikidata) [production]
17:02 <ema> depooled ulsfo https://phabricator.wikimedia.org/T127094 [production]
16:48 <ostriches> purged ancient boardvote gpg key from mediawiki fleet. unused since forever. [production]
16:25 <anomie@tin> Synchronized wmf-config/: SWAT: Undeploy Extension:ApiSandbox (duration: 01m 30s) [production]
16:20 <anomie@tin> Synchronized wmf-config/CommonSettings.php: SWAT: Remove $wgMWOAuthGrantPermissions (duration: 01m 34s) [production]
16:16 <urandom> restbase deploy (15a6c50) complete, sans restbase1008.eqiad.wmnet (down for maintenance during deploy) [production]
16:16 <anomie> Ran namespaceDupes.php on tawiki [production]
16:14 <urandom> restbase deploy (15a6c50) completet [production]
16:14 <hoo> Re-populating the sites and site_identifiers table for all Wikipedias and testwikidata [production]
16:10 <urandom> restbase deploy restarting at restbase1009 [production]
16:09 <anomie@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: New userrights and configuration for cswiki (task [[phab:T126931|]]) (duration: 01m 31s) [production]
16:09 <urandom> restbase deploy stalled at restbase1008 (under maintenance) [production]
16:05 <anomie@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace aliases for tawiki (task [[phab:T126604|]]) (duration: 01m 31s) [production]
16:00 <urandom> continuing production-wide restbase deploy (15a6c50) [production]
15:58 <godog> copy restbase.{v1_*,sys_*,ALL,GET,HEAD,POST,OPTIONS,_robots} to restbase.external on graphite1001 and graphite2001 [production]
15:55 <godog> copy restbase.private to restbase.internal on graphite1001 and graphite2001 [production]
15:53 <aude@tin> Synchronized php-1.27.0-wmf.13/extensions/Wikidata: Fix caching data types bug: T127095 (duration: 01m 44s) [production]
15:53 <urandom> canary deploy of restbase to restbase1001.eqiad.wmnet (15a6c50) complete [production]
15:53 <urandom> canary deploy of restbase to restbase1001.eqiad.wmnet (15a6c50) [production]
15:51 <bblack> package upgrades commencing on lvs* [production]
15:43 <urandom> restbase staging deploy (15a6c50) complete [production]