|
2016-01-21
§
|
| 12:10 |
<paravoid> |
upgrading cr1-codfw to JunOS 13.3R8.7 |
[production] |
| 11:27 |
<_joe_> |
restarting pybal on lvs4003, switching to etcd |
[production] |
| 11:25 |
<_joe_> |
restarting pybal on lvs4004, switching to etcd |
[production] |
| 11:09 |
<jynus> |
adding new version of mariadb to carbon for jessie (10.0.23-1) |
[production] |
| 10:19 |
<_joe_> |
mw2098 doesn't reboot, console unreachable |
[production] |
| 10:10 |
<jynus> |
mw2098.codfw.wmnet failed to sync |
[production] |
| 10:10 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Restore s5 DB configuration (duration: 01m 57s) |
[production] |
| 09:53 |
<_joe_> |
rolling reboot of the codfw appserver layer |
[production] |
| 09:27 |
<_joe_> |
powercycled mw1162, memory exhaustion |
[production] |
| 08:01 |
<_joe_> |
upgrading all codfw appserver layer's kernel to linux-image-3.13.0-76-generic |
[production] |
| 02:56 |
<l10nupdate@tin> |
l10nupdate@tin ResourceLoader cache refresh completed at Thu Jan 21 02:56:44 UTC 2016 (duration 7m 9s) |
[production] |
| 02:49 |
<mwdeploy@tin> |
mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 09m 39s) |
[production] |
| 02:27 |
<mwdeploy@tin> |
mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 09m 33s) |
[production] |
| 02:24 |
<mobrovac> |
citoid deploying 3a1b6c8648 |
[production] |
| 02:16 |
<ori> |
Restarting jobrunner service on job runners to ensure I180856917 gets picked up |
[production] |
| 01:47 |
<mutante> |
nitrogen - install package upgrades |
[production] |
| 01:15 |
<bd808> |
Restarted logstash on logstash1003 |
[production] |
| 01:14 |
<bd808> |
Restarted logstash on logstash1002 |
[production] |
| 01:04 |
<maxsem@tin> |
maxsem@tin Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/265395/ (duration: 00m 32s) |
[production] |
| 00:56 |
<maxsem@tin> |
maxsem@tin Synchronized php-1.27.0-wmf.11/extensions/GeoData/: https://gerrit.wikimedia.org/r/#/c/265409/ (duration: 00m 33s) |
[production] |
| 00:50 |
<maxsem@tin> |
maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/265142/ (duration: 00m 32s) |
[production] |
|
2016-01-20
§
|
| 23:56 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.10/extensions/SemanticForms/: fix wikitech again (duration: 00m 34s) |
[production] |
| 23:06 |
<bd808> |
Restarted logstash on logstash1001 |
[production] |
| 23:04 |
<bd808> |
Logstash1001 went nuts and decided that instead of 2016 it would go back to the start of 2015 after 2015-12-31T23:59 |
[production] |
| 22:54 |
<bd808> |
no HHVM log events in logstash since 2015-12-31T23:59:44.000Z |
[production] |
| 22:48 |
<bd808> |
HHVM log messages not being recorded in Logstash; bd808 to investigate |
[production] |
| 22:38 |
<tgr@tin> |
tgr@tin Synchronized php-1.27.0-wmf.11/includes/: T124143,T124126 (duration: 00m 36s) |
[production] |
| 22:06 |
<anomie@tin> |
anomie@tin Synchronized php-1.27.0-wmf.11/extensions/OAuth: Deploy fix for T124224 (duration: 00m 32s) |
[production] |
| 22:04 |
<anomie@tin> |
anomie@tin Synchronized php-1.27.0-wmf.2/extensions/OAuth: Deploy fix for T124224 (duration: 00m 34s) |
[production] |
| 21:51 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticResultFormats: Fix wikitech log noise (duration: 00m 31s) |
[production] |
| 21:50 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticMediaWiki: Fix wikitech log noise (duration: 00m 34s) |
[production] |
| 21:48 |
<subbu> |
finished deploying parsoid sha f1ddfb88 |
[production] |
| 21:41 |
<subbu> |
synced new parsoid code; restarted parsoid on wtp1001 as a canary |
[production] |
| 21:35 |
<subbu> |
starting parsoid deploy |
[production] |
| 21:32 |
<thcipriani> |
reverted group1 wikis to 1.27.0-wmf.10 due to session errors. |
[production] |
| 21:30 |
<thcipriani@tin> |
thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.10 |
[production] |
| 21:14 |
<andrewbogott> |
rebooting labvirt1011 |
[production] |
| 21:08 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticForms/: Fix fatal on wikitech (duration: 00m 36s) |
[production] |
| 20:37 |
<akosiaris> |
s#/dev/md1#/dev/mapper/tank-data# on labvirt1010, reverted by puppet with Notice: /Stage[main]/Role::Labs::Openstack::Nova::Compute/Mount[/var/lib/nova/instances]/device: device changed '/dev/mapper/tank-data' to '/dev/md1' |
[production] |
| 20:37 |
<akosiaris> |
s#/dev/md1#/dev/mapper/tank-data# |
[production] |
| 19:32 |
<dduvall@tin> |
dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.11 |
[production] |
| 19:13 |
<marxarelli> |
including labswiki and labtestwiki in group1 promotion after all |
[production] |
| 19:09 |
<marxarelli> |
starting promotion of group1, but holding back labswiki and labtestwiki until Jan 21 'all' promotion |
[production] |
| 18:54 |
<paravoid> |
manually triggering an ubuntu mirror update ("sudo -u mirror /usr/local/sbin/update-ubuntu-mirror" on carbon) |
[production] |
| 18:41 |
<jynus> |
schema change on wikidatawiki (wb_terms) finished- slaves already catching up |
[production] |
| 18:34 |
<mutante> |
restart hhvm on mw1206 |
[production] |
| 18:32 |
<godog> |
bounce stuck hhvm on mw1205 |
[production] |
| 18:06 |
<paravoid> |
turning up BGP with Zayo in codfw |
[production] |
| 17:48 |
<jynus> |
restarting replication on db1026 after schema change |
[production] |
| 17:09 |
<gwicke> |
restbase cassandra: set DTCS max_window_size_seconds to 70736000, large enough to accommodate a two-year window |
[production] |