|
2016-01-20
§
|
| 23:56 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.10/extensions/SemanticForms/: fix wikitech again (duration: 00m 34s) |
[production] |
| 23:06 |
<bd808> |
Restarted logstash on logstash1001 |
[production] |
| 23:04 |
<bd808> |
Logstash1001 went nuts and decided that instead of 2016 it would go back to the start of 2015 after 2015-12-31T23:59 |
[production] |
| 22:54 |
<bd808> |
no HHVM log events in logstash since 2015-12-31T23:59:44.000Z |
[production] |
| 22:48 |
<bd808> |
HHVM log messages not being recorded in Logstash; bd808 to investigate |
[production] |
| 22:38 |
<tgr@tin> |
tgr@tin Synchronized php-1.27.0-wmf.11/includes/: T124143,T124126 (duration: 00m 36s) |
[production] |
| 22:36 |
<bd808> |
Seems to be working again; possible intermittent Labs dns issues |
[tools.bash] |
| 22:34 |
<bd808> |
Can't resolve host issues contacting elasticsearch backend |
[tools.bash] |
| 22:33 |
<bd808> |
Webservice returning 404 before restart, 502 after restart; not usable log data |
[tools.bash] |
| 22:06 |
<anomie@tin> |
anomie@tin Synchronized php-1.27.0-wmf.11/extensions/OAuth: Deploy fix for T124224 (duration: 00m 32s) |
[production] |
| 22:04 |
<anomie@tin> |
anomie@tin Synchronized php-1.27.0-wmf.2/extensions/OAuth: Deploy fix for T124224 (duration: 00m 34s) |
[production] |
| 21:51 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticResultFormats: Fix wikitech log noise (duration: 00m 31s) |
[production] |
| 21:50 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticMediaWiki: Fix wikitech log noise (duration: 00m 34s) |
[production] |
| 21:48 |
<subbu> |
finished deploying parsoid sha f1ddfb88 |
[production] |
| 21:41 |
<subbu> |
synced new parsoid code; restarted parsoid on wtp1001 as a canary |
[production] |
| 21:35 |
<subbu> |
starting parsoid deploy |
[production] |
| 21:32 |
<thcipriani> |
reverted group1 wikis to 1.27.0-wmf.10 due to session errors. |
[production] |
| 21:30 |
<thcipriani@tin> |
thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.10 |
[production] |
| 21:14 |
<andrewbogott> |
rebooting labvirt1011 |
[production] |
| 21:08 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticForms/: Fix fatal on wikitech (duration: 00m 36s) |
[production] |
| 20:37 |
<akosiaris> |
s#/dev/md1#/dev/mapper/tank-data# on labvirt1010, reverted by puppet with Notice: /Stage[main]/Role::Labs::Openstack::Nova::Compute/Mount[/var/lib/nova/instances]/device: device changed '/dev/mapper/tank-data' to '/dev/md1' |
[production] |
| 20:37 |
<akosiaris> |
s#/dev/md1#/dev/mapper/tank-data# |
[production] |
| 20:05 |
<hashar> |
beta sudo find /data/project/upload7/math -type f -delete (probably some old left over) |
[releng] |
| 19:50 |
<hashar> |
beta: on commons ran deleteArchivedFile.php : Nuked 7130 files |
[releng] |
| 19:49 |
<hashar> |
beta : foreachwiki deleteArchivedRevisions.php -delete |
[releng] |
| 19:32 |
<dduvall@tin> |
dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.11 |
[production] |
| 19:26 |
<hasharAway> |
Nuked all files from http://commons.wikimedia.beta.wmflabs.org/wiki/Category:GWToolset_Batch_Upload |
[releng] |
| 19:19 |
<hasharAway> |
beta: sudo find /data/project/upload7/*/*/temp -type f -delete |
[releng] |
| 19:14 |
<hasharAway> |
beta: sudo rm /data/project/upload7/*/*/lockdir/* |
[releng] |
| 19:13 |
<marxarelli> |
including labswiki and labtestwiki in group1 promotion after all |
[production] |
| 19:09 |
<marxarelli> |
starting promotion of group1, but holding back labswiki and labtestwiki until Jan 21 'all' promotion |
[production] |
| 18:56 |
<hasharAway> |
beta cluster code has been stalled for roughly 2h30 |
[releng] |
| 18:55 |
<hasharAway> |
disconnecting Gearman plugin to remove deadlock for beta cluster rjobs |
[releng] |
| 18:54 |
<paravoid> |
manually triggering an ubuntu mirror update ("sudo -u mirror /usr/local/sbin/update-ubuntu-mirror" on carbon) |
[production] |
| 18:41 |
<jynus> |
schema change on wikidatawiki (wb_terms) finished- slaves already catching up |
[production] |
| 18:34 |
<mutante> |
restart hhvm on mw1206 |
[production] |
| 18:32 |
<godog> |
bounce stuck hhvm on mw1205 |
[production] |
| 18:30 |
<mforns> |
deployed EL in production with removal of queue |
[analytics] |
| 18:06 |
<paravoid> |
turning up BGP with Zayo in codfw |
[production] |
| 17:48 |
<jynus> |
restarting replication on db1026 after schema change |
[production] |
| 17:37 |
<mforns> |
restarted EventLogging because of Kafka consumption lag |
[analytics] |
| 17:09 |
<gwicke> |
restbase cassandra: set DTCS max_window_size_seconds to 70736000, large enough to accommodate a two-year window |
[production] |