3101-3150 of 10000 results (25ms)
2016-03-28 §
08:21 <paravoid> powercycling ms-be2016, down for ~1d, unresponsive on console [production]
02:35 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Mar 28 02:35:00 UTC 2016 (duration 8m 26s) [production]
02:26 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.18) (duration: 11m 32s) [production]
2016-03-27 §
22:21 <ori@tin> Synchronized wmf-config/StartProfiler.php: I1b5c620b85: Better request profiling via XWD header (duration: 00m 33s) [production]
20:11 <elukey> restarted hhvm on mw1123 [production]
20:04 <elukey> Increased nf_conntrack_max to ~528k for the kafka brokers (https://gerrit.wikimedia.org/r/279776) [production]
02:57 <jzerebecki> disabling 2fa for Dereckson T130892 [labswiki]> delete from oathauth_users where id=402; [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Mar 27 02:31:03 UTC 2016 (duration 8m 35s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.18) (duration: 10m 06s) [production]
01:51 <jzerebecki> disabling 2fa for Hym411 T130994 [labswiki]> delete from oathauth_users where id=1363; [production]
2016-03-26 §
22:38 <jzerebecki> reloading zuul for 2d7e050..46923c8 [releng]
21:11 <csteipp_ooo> removed 2FA from wikitech accounts that looked to be affected by T130892 [production]
16:44 <Luke081515> updating all git repos [rcm.cac]
16:21 <krenair@tin> Synchronized php-1.27.0-wmf.18/extensions/SemanticForms: https://gerrit.wikimedia.org/r/#/c/279701/ (duration: 00m 32s) [production]
13:46 <urandom> Cassandra on restbase2004.codfw.wmnet shut down, hardware failure; Down for the weekend : T130990 [production]
13:26 <urandom> Stopping Cassandra on restbase2004.codfw.wmnet [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Mar 26 02:31:13 UTC 2016 (duration 8m 34s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.18) (duration: 10m 06s) [production]
00:11 <mutante|away> scb[12]00[12] - delete changeprop main.log per request [production]
2016-03-25 §
23:57 <bd808> Removed self (BryanDavis) from project [xtools]
23:56 <bd808> Deleted stale puppet lock file on xtools-lb [xtools]
23:55 <marxarelli> deleting instances integration-slave-trusty-1002 and integration-slave-trusty-1005 [releng]
23:54 <bd808> Added self (BryanDavis) as admin to debug puppet failure [xtools]
23:54 <marxarelli> deleting jenkins nodes integration-slave-trusty-1002 and integration-slave-trusty-1005 [releng]
23:41 <marxarelli> completed rolling manual deploy of https://gerrit.wikimedia.org/r/#/c/279640/ to trusty slaves [releng]
23:27 <marxarelli> starting rolling offline/remount/online of trusty slaves to increase tmpfs size [releng]
23:22 <marxarelli> pooled new trusty slaves integration-slave-trusty-1024 and integration-slave-trusty-1025 [releng]
23:13 <jzerebecki> reloading zuul fro 0aec21d..2d7e050 [releng]
22:14 <marxarelli> creating new jenkins node for integration-slave-trusty-1024 [releng]
22:11 <marxarelli> rebooting integration-slave-trusty-{1024,1025} before pooling as replacements for trusty-1002 and trusty-1005 [releng]
21:52 <ori> Updated Grafana to latest nightly [production]
21:19 <ori@tin> Synchronized wmf-config/logging.php: Ibc96f9d3bd: Integrate X-Wikimedia-Debug with Logstash; I28b137f5e035: Logging: convert to short array syntax (duration: 00m 36s) [production]
21:06 <marxarelli> repooling integration-slave-trusty-{1005,1002} to help with load while replacement instances are provisioning [releng]
20:46 <ori@tin> Synchronized php-1.27.0-wmf.18/includes/filerepo/LocalRepo.php: Idaa1237638: Request-local caching of image_redirect and I545ce6b160b: Lower pcTTL in checkRedirect() to 30 (duration: 00m 37s) [production]
20:23 <Krenair> started redis-server on deployment-redis01 [deployment-prep]
20:23 <Krenair> repaired centralauth.spoofuser table on deployment-db1 [deployment-prep]
20:23 <Krenair> fiddled around with puppet on deployment-cache-text04 earlier to fix certs etc. [deployment-prep]
19:31 <tgr> running extensions/CentralAuth/maintenance/checkLocal{Names,User}.php on jawiktionary and dewikiquote [production]
19:08 <mutante> scb1001 - Unit changeprop.service entered failed state - it's a new deployment though, acked earlier by mobrovac already [production]
18:27 <mobrovac> restbase started mobile-sections dump of enwiki on restbase1009 for articles edited before 2016-03-23 as per T130698 [production]
17:19 <akosiaris> rolling restart of restbase nodes to apply https://gerrit.wikimedia.org/r/#/c/279597/ [production]
16:59 <marxarelli> depooling integration-slave-trusty-1002 until DNS resolution can be resolved. still investigating disk space issue [releng]
16:55 <ori@tin> Synchronized wmf-config/logging.php: Ia2cd5daaf3: Update logging config for request IDs (duration: 00m 31s) [production]
16:50 <ori@tin> Synchronized php-1.27.0-wmf.18/includes: Iaf90c20c330e: Provide a unique request identifier (duration: 01m 22s) [production]
15:32 <demon@tin> Synchronized wmf-config/InitialiseSettings.php: Remove T44894 FIXME note (duration: 00m 27s) [production]
15:03 <demon@tin> Synchronized wmf-config/InitialiseSettings.php: Remove sampling from ApiAction kafka channel (duration: 00m 33s) [production]
14:58 <demon@tin> Synchronized wmf-config/db-codfw.php: docfix (duration: 00m 34s) [production]
14:43 <godog> chown _graphite:_graphite frontend.navtiming.loadEventEnd [production]
14:38 <godog> depool restbase and drain cassandra from restbase1007 [production]
14:20 <bblack> all authdns servers running ferm rules now [production]