1651-1700 of 10000 results (31ms)
2016-01-21 §
11:09 <jynus> adding new version of mariadb to carbon for jessie (10.0.23-1) [production]
10:19 <_joe_> mw2098 doesn't reboot, console unreachable [production]
10:10 <jynus> mw2098.codfw.wmnet failed to sync [production]
10:10 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Restore s5 DB configuration (duration: 01m 57s) [production]
09:53 <_joe_> rolling reboot of the codfw appserver layer [production]
09:27 <_joe_> powercycled mw1162, memory exhaustion [production]
08:01 <_joe_> upgrading all codfw appserver layer's kernel to linux-image-3.13.0-76-generic [production]
02:56 <l10nupdate@tin> l10nupdate@tin ResourceLoader cache refresh completed at Thu Jan 21 02:56:44 UTC 2016 (duration 7m 9s) [production]
02:49 <mwdeploy@tin> mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 09m 39s) [production]
02:27 <mwdeploy@tin> mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 09m 33s) [production]
02:24 <mobrovac> citoid deploying 3a1b6c8648 [production]
02:16 <ori> Restarting jobrunner service on job runners to ensure I180856917 gets picked up [production]
01:47 <mutante> nitrogen - install package upgrades [production]
01:15 <bd808> Restarted logstash on logstash1003 [production]
01:14 <bd808> Restarted logstash on logstash1002 [production]
01:04 <maxsem@tin> maxsem@tin Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/265395/ (duration: 00m 32s) [production]
00:56 <maxsem@tin> maxsem@tin Synchronized php-1.27.0-wmf.11/extensions/GeoData/: https://gerrit.wikimedia.org/r/#/c/265409/ (duration: 00m 33s) [production]
00:50 <maxsem@tin> maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/265142/ (duration: 00m 32s) [production]
2016-01-20 §
23:56 <reedy@tin> reedy@tin Synchronized php-1.27.0-wmf.10/extensions/SemanticForms/: fix wikitech again (duration: 00m 34s) [production]
23:06 <bd808> Restarted logstash on logstash1001 [production]
23:04 <bd808> Logstash1001 went nuts and decided that instead of 2016 it would go back to the start of 2015 after 2015-12-31T23:59 [production]
22:54 <bd808> no HHVM log events in logstash since 2015-12-31T23:59:44.000Z [production]
22:48 <bd808> HHVM log messages not being recorded in Logstash; bd808 to investigate [production]
22:38 <tgr@tin> tgr@tin Synchronized php-1.27.0-wmf.11/includes/: T124143,T124126 (duration: 00m 36s) [production]
22:06 <anomie@tin> anomie@tin Synchronized php-1.27.0-wmf.11/extensions/OAuth: Deploy fix for T124224 (duration: 00m 32s) [production]
22:04 <anomie@tin> anomie@tin Synchronized php-1.27.0-wmf.2/extensions/OAuth: Deploy fix for T124224 (duration: 00m 34s) [production]
21:51 <reedy@tin> reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticResultFormats: Fix wikitech log noise (duration: 00m 31s) [production]
21:50 <reedy@tin> reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticMediaWiki: Fix wikitech log noise (duration: 00m 34s) [production]
21:48 <subbu> finished deploying parsoid sha f1ddfb88 [production]
21:41 <subbu> synced new parsoid code; restarted parsoid on wtp1001 as a canary [production]
21:35 <subbu> starting parsoid deploy [production]
21:32 <thcipriani> reverted group1 wikis to 1.27.0-wmf.10 due to session errors. [production]
21:30 <thcipriani@tin> thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.10 [production]
21:14 <andrewbogott> rebooting labvirt1011 [production]
21:08 <reedy@tin> reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticForms/: Fix fatal on wikitech (duration: 00m 36s) [production]
20:37 <akosiaris> s#/dev/md1#/dev/mapper/tank-data# on labvirt1010, reverted by puppet with Notice: /Stage[main]/Role::Labs::Openstack::Nova::Compute/Mount[/var/lib/nova/instances]/device: device changed '/dev/mapper/tank-data' to '/dev/md1' [production]
20:37 <akosiaris> s#/dev/md1#/dev/mapper/tank-data# [production]
19:32 <dduvall@tin> dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.11 [production]
19:13 <marxarelli> including labswiki and labtestwiki in group1 promotion after all [production]
19:09 <marxarelli> starting promotion of group1, but holding back labswiki and labtestwiki until Jan 21 'all' promotion [production]
18:54 <paravoid> manually triggering an ubuntu mirror update ("sudo -u mirror /usr/local/sbin/update-ubuntu-mirror" on carbon) [production]
18:41 <jynus> schema change on wikidatawiki (wb_terms) finished- slaves already catching up [production]
18:34 <mutante> restart hhvm on mw1206 [production]
18:32 <godog> bounce stuck hhvm on mw1205 [production]
18:06 <paravoid> turning up BGP with Zayo in codfw [production]
17:48 <jynus> restarting replication on db1026 after schema change [production]
17:09 <gwicke> restbase cassandra: set DTCS max_window_size_seconds to 70736000, large enough to accommodate a two-year window [production]
16:56 <thcipriani@tin> thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set default graph vega version back to 1 [[gerrit:265289]] (duration: 00m 32s) [production]
16:46 <thcipriani@tin> thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add davidabian.com to wgCopyUploadsDomains [[gerrit:265286]] (duration: 00m 32s) [production]
16:42 <thcipriani@tin> thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Change default graph version param. Part II [[gerrit:265282]] (duration: 00m 32s) [production]