9251-9300 of 10000 results (46ms)
2016-03-28 §
13:32 <bblack> rebooting radon (unresponsive console) [production]
08:44 <paravoid> force-rebooting ms-be2008 to fix hundreds of unkillable mkfs.xfs stuck due to a hosed disk [production]
08:21 <paravoid> powercycling ms-be2016, down for ~1d, unresponsive on console [production]
02:35 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Mar 28 02:35:00 UTC 2016 (duration 8m 26s) [production]
02:26 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.18) (duration: 11m 32s) [production]
2016-03-27 §
22:21 <ori@tin> Synchronized wmf-config/StartProfiler.php: I1b5c620b85: Better request profiling via XWD header (duration: 00m 33s) [production]
20:11 <elukey> restarted hhvm on mw1123 [production]
20:04 <elukey> Increased nf_conntrack_max to ~528k for the kafka brokers (https://gerrit.wikimedia.org/r/279776) [production]
02:57 <jzerebecki> disabling 2fa for Dereckson T130892 [labswiki]> delete from oathauth_users where id=402; [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Mar 27 02:31:03 UTC 2016 (duration 8m 35s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.18) (duration: 10m 06s) [production]
01:51 <jzerebecki> disabling 2fa for Hym411 T130994 [labswiki]> delete from oathauth_users where id=1363; [production]
2016-03-26 §
21:11 <csteipp_ooo> removed 2FA from wikitech accounts that looked to be affected by T130892 [production]
16:21 <krenair@tin> Synchronized php-1.27.0-wmf.18/extensions/SemanticForms: https://gerrit.wikimedia.org/r/#/c/279701/ (duration: 00m 32s) [production]
13:46 <urandom> Cassandra on restbase2004.codfw.wmnet shut down, hardware failure; Down for the weekend : T130990 [production]
13:26 <urandom> Stopping Cassandra on restbase2004.codfw.wmnet [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Mar 26 02:31:13 UTC 2016 (duration 8m 34s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.18) (duration: 10m 06s) [production]
00:11 <mutante|away> scb[12]00[12] - delete changeprop main.log per request [production]
2016-03-25 §
21:52 <ori> Updated Grafana to latest nightly [production]
21:19 <ori@tin> Synchronized wmf-config/logging.php: Ibc96f9d3bd: Integrate X-Wikimedia-Debug with Logstash; I28b137f5e035: Logging: convert to short array syntax (duration: 00m 36s) [production]
20:46 <ori@tin> Synchronized php-1.27.0-wmf.18/includes/filerepo/LocalRepo.php: Idaa1237638: Request-local caching of image_redirect and I545ce6b160b: Lower pcTTL in checkRedirect() to 30 (duration: 00m 37s) [production]
19:31 <tgr> running extensions/CentralAuth/maintenance/checkLocal{Names,User}.php on jawiktionary and dewikiquote [production]
19:08 <mutante> scb1001 - Unit changeprop.service entered failed state - it's a new deployment though, acked earlier by mobrovac already [production]
18:27 <mobrovac> restbase started mobile-sections dump of enwiki on restbase1009 for articles edited before 2016-03-23 as per T130698 [production]
17:19 <akosiaris> rolling restart of restbase nodes to apply https://gerrit.wikimedia.org/r/#/c/279597/ [production]
16:55 <ori@tin> Synchronized wmf-config/logging.php: Ia2cd5daaf3: Update logging config for request IDs (duration: 00m 31s) [production]
16:50 <ori@tin> Synchronized php-1.27.0-wmf.18/includes: Iaf90c20c330e: Provide a unique request identifier (duration: 01m 22s) [production]
15:32 <demon@tin> Synchronized wmf-config/InitialiseSettings.php: Remove T44894 FIXME note (duration: 00m 27s) [production]
15:03 <demon@tin> Synchronized wmf-config/InitialiseSettings.php: Remove sampling from ApiAction kafka channel (duration: 00m 33s) [production]
14:58 <demon@tin> Synchronized wmf-config/db-codfw.php: docfix (duration: 00m 34s) [production]
14:43 <godog> chown _graphite:_graphite frontend.navtiming.loadEventEnd [production]
14:38 <godog> depool restbase and drain cassandra from restbase1007 [production]
14:20 <bblack> all authdns servers running ferm rules now [production]
13:15 <bblack> enabling ferm on authdns servers (one at a time, while watching stuff...) [production]
08:38 <akosiaris> depool maps-test2002.codfw.wmnet temporarily [production]
03:45 <csteipp> redeployed Scribunto patch for T110143 [production]
02:38 <l10nupdate@tin> ResourceLoader cache refresh completed at Fri Mar 25 02:38:53 UTC 2016 (duration 8m 55s) [production]
02:29 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.18) (duration: 09m 54s) [production]
00:24 <catrope@tin> Finished scap: Pushing out i18n changes from SWAT (duration: 26m 15s) [production]
2016-03-24 §
23:58 <catrope@tin> Started scap: Pushing out i18n changes from SWAT [production]
23:52 <catrope@tin> Synchronized php-1.27.0-wmf.18/extensions/ContentTranslation: Remove registration of missing i18n keys (duration: 00m 28s) [production]
23:51 <catrope@tin> Synchronized php-1.27.0-wmf.18/extensions/Flow: Remove registration of missing i18n keys (duration: 00m 46s) [production]
23:51 <catrope@tin> Synchronized php-1.27.0-wmf.18/extensions/Graph: Remove registration of missing i18n keys (duration: 00m 28s) [production]
23:42 <catrope@tin> Synchronized php-1.27.0-wmf.18/extensions/MobileApp/: Roll out RESTbase usage to mobile app at 2% (duration: 00m 27s) [production]
23:38 <catrope@tin> Synchronized php-1.27.0-wmf.18/extensions/MobileFrontend/: SWAT (duration: 00m 30s) [production]
23:31 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Enable Echo footer notice on enwiki (duration: 00m 28s) [production]
23:22 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Switch back to legacy language overlay (duration: 00m 27s) [production]
23:16 <catrope@tin> Synchronized php-1.27.0-wmf.18/extensions/Echo/: Styling fixes for footer link (duration: 00m 31s) [production]
23:15 <catrope@tin> Synchronized php-1.27.0-wmf.18/extensions/OATHAuth/: SWAT (duration: 00m 28s) [production]