5651-5700 of 10000 results (46ms)
2016-05-03 §
14:36 <gehel> restarting elasticsearch server elastic2007.codfw.wmnet (T110236) [production]
14:12 <elukey> changed kafkatee's config on oxygen to watch all the webrequest text|upload kafka partitions (analytics doubled them recently) [production]
14:00 <jynus> creating sanitarium db filtering (redactatron) for jamwiki [production]
13:51 <hashar> Warming up HHVM cache hitting testwiki /w/api.php T131557 [production]
13:40 <gehel> restarting elasticsearch server elastic2006.codfw.wmnet (T110236) [production]
13:38 <hashar> Warming up HHVM cache hitting testwiki T131557 [production]
13:38 <hashar> Warming up HHVM cache hitting testwiki T1315567 [production]
13:37 <hashar@tin> Finished scap: testwiki to php-1.27.0-wmf.23 and rebuild l10n cache T131557 (duration: 26m 54s) [production]
13:18 <jynus> applying schema change on s3-master db T130692 [production]
13:10 <hashar@tin> Started scap: testwiki to php-1.27.0-wmf.23 and rebuild l10n cache T131557 [production]
13:02 <gehel> restarting elasticsearch server elastic2005.codfw.wmnet (T110236) [production]
13:01 <gehel> restarting wdqs-updater and keeping it under close scrutiny for the moment [production]
13:00 <volans> Change runtime semi-synchronous replication on eqiad core DBs (s1-s7) to match configured value T131753 [production]
12:31 <gehel> restarting elasticsearch server elastic2004.codfw.wmnet (T110236) [production]
12:28 <gehel> stopping wdqs-updater as it leaks pipes [production]
12:07 <hashar> Cutting 1.27.0-wmf.23 branches T131557 [production]
12:07 <hashar> Cutting 1.27.0-wmf.23 branches [production]
11:35 <bblack> cp1068: upgraded varnish to 3.0.6plus-wm9 [production]
10:42 <moritzm> installing poppler security updates on ocg (and other trusty hosts) [production]
10:24 <jynus> applying schema change on eqiad-s3 db servers T130692 [production]
10:03 <jynus> applying schema change on codfw-s3 db servers T130692 [production]
09:55 <jynus> testing online index creation on db2018 [production]
09:11 <gehel> restarting elasticsearch server elastic2003.codfw.wmnet (T110236) [production]
09:10 <moritzm> powercycled alsafi (stuck in KVM) [production]
09:08 <kart_> Update cxserver to 8a4254e [production]
08:18 <gehel> restarting elasticsearch server elastic2002.codfw.wmnet (T110236) [production]
08:05 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1040 after maintenance (duration: 00m 25s) [production]
07:06 <jynus> recovering torrus database @ netmon following T87815 [production]
02:34 <l10nupdate@tin> ResourceLoader cache refresh completed at Tue May 3 02:34:45 UTC 2016 (duration 8m 36s) [production]
02:26 <awight> CentralNotice fundraising campaigns reenabled after CiviCRM maintenance [production]
02:26 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.22) (duration: 08m 40s) [production]
02:21 <mutante> ssh alsafi [production]
02:13 <eileen> Updating civicrm from f5e8f98d07a2280118b7153bc342bf52ee67edd5 to d32032965f2a45d24f1eaeb352eb6bfd6641753c [production]
02:05 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Remove obsolete https->http rewrite for IRC notifications (duration: 00m 28s) [production]
02:04 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Remove obsolete ocwiki hack (duration: 00m 37s) [production]
01:54 <RoanKattouw> Killed ssh processes on tin that had been hanging for days [production]
01:53 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.21) (duration: 23019m 56s) [production]
01:53 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.21) (duration: 12941m 41s) [production]
01:53 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.22) (duration: 4303m 19s) [production]
01:40 <mutante> irc.wm.org - see T123729 if any questions [production]
01:39 <mutante> switching irc.wikimedia.org from old server argon to new server kraz. old server still running untouched as argon.wikimedia.org. no clients are kicked. appservers are sending RC to both. [production]
00:50 <krenair@tin> Synchronized wmf-config/interwiki.php: https://gerrit.wikimedia.org/r/286552 - interwiki cache update including jamwiki, horizon, and other things (duration: 00m 26s) [production]
00:41 <krenair@tin> Synchronized langlist: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 24s) [production]
00:40 <krenair@tin> Synchronized static/images/project-logos: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 24s) [production]
00:39 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 24s) [production]
00:38 <krenair@tin> rebuilt wikiversions.php and synchronized wikiversions files: create jamwiki - https://gerrit.wikimedia.org/r/286258 [production]
00:36 <krenair@tin> Synchronized dblists: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 27s) [production]
00:12 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/286534/ (duration: 00m 30s) [production]
2016-05-02 §
23:58 <krenair@tin> Finished scap: for https://gerrit.wikimedia.org/r/#/c/286535/ i18n changes (duration: 25m 57s) [production]
23:37 <ebernhardson> restart elastic2007, codfw cluster master, to resolve lingering issues after resolving frozen index race condition [production]