|
2015-10-23
§
|
| 23:31 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki hewikisource |
[production] |
| 23:31 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki ukwiki |
[production] |
| 23:09 |
<hoo> |
Started a local rename for Stefan4 to Stefan2 on commons, per request (and attributed to) DerHexer |
[production] |
| 23:05 |
<marxarelli> |
restoring deployment-db2 from dump |
[releng] |
| 23:03 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/248551 |
[releng] |
| 22:23 |
<robh> |
powercycled nobelium |
[production] |
| 22:18 |
<marxarelli> |
dump of deployment-db1 failed due to "View 'labswiki.bounce_records' references invalid table(s)" |
[releng] |
| 21:37 |
<marxarelli> |
dumping databases on deployment-db1 for restore of deployment-db2 |
[releng] |
| 21:24 |
<mutante> |
nobelium - powercycled, no console output |
[production] |
| 21:13 |
<marxarelli> |
deployment-db1 binlog deployment-db1-bin.000062 appears corrupt |
[releng] |
| 20:54 |
<marxarelli> |
deployment-db2 shows slave io but slave sql failed on duplicate key |
[releng] |
| 20:04 |
<jynus> |
rename user job had created lag on almost all enwiki dbs, things should be better now |
[production] |
| 19:08 |
<urandom> |
starting nodetool cleanup on restbase-test2001-b |
[production] |
| 19:07 |
<urandom> |
starting nodetool cleanup on restbase-test2001-a |
[production] |
| 18:59 |
<twentyafterfour> |
deleted atop.log.* files on deployment-bastion. when are we going to enlarge /var on this instance. grr |
[releng] |
| 18:58 |
<marxarelli> |
Killed mysql process 15840440 on account of its gargantuan temp file filling up /mnt |
[releng] |
| 18:16 |
<godog> |
bounce grafana-server on krypton |
[production] |
| 17:58 |
<YuviPanda> |
moved mounts around on nobelium, mounted bigger disk on /var/lib/elasticsearch |
[production] |
| 17:57 |
<godog> |
delete wikimedia-grid grafana dashboard (saved a copy first) heavy graphite queries |
[production] |
| 17:53 |
<YuviPanda> |
stopping elasticsearch on nobelium for https://phabricator.wikimedia.org/T114856 |
[production] |
| 17:13 |
<akosiaris> |
enable_notifications=0 in neon's icinga for a few mins while the storm dies down |
[production] |
| 16:47 |
<godog> |
stop puppet on graphite1001 and graphite-index cron, suspected root cause |
[production] |
| 16:26 |
<godog> |
uwsgi timing out while serving requests, bounce also carbon daemons |
[production] |
| 16:20 |
<godog> |
bounce uwsgi for graphite-web on graphite1001 |
[production] |
| 16:14 |
<urandom> |
starting rebuild of restbase-test2002-b |
[production] |
| 15:42 |
<jynus> |
copied wmf-mariadb10 (10.0.16-2) .deb from trusty to jessie on apt.wikimedia.org |
[production] |
| 15:42 |
<urandom> |
bouncing restbase-test2002-a Just To See |
[production] |
| 15:06 |
<urandom> |
running nodetool cleanup on restbase-test2003 |
[production] |
| 14:47 |
<godog> |
remove outdated cassandra metrics from graphite2001 |
[production] |
| 14:27 |
<godog> |
remove restbase-test2001 restbase-test2002 cassandra metrics |
[production] |
| 14:12 |
<mobrovac> |
restbase rolling-restarting after applying aab840f |
[production] |
| 13:55 |
<jynus> |
Rebooting and installing jessie on db2055-db2059 |
[production] |
| 13:45 |
<godog> |
remove 98percentile 999percentile meanRate stddev cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/ |
[production] |
| 13:33 |
<godog> |
remove 15MinuteRate cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/ |
[production] |
| 11:51 |
<godog> |
roll-restart cassandra-metrics-collector on restbase cluster after https://gerrit.wikimedia.org/r/248313 |
[production] |
| 11:14 |
<jynus> |
After days of no references on the logs to the renamed user_daily_contribs, deleting delete_user_daily_contribs table on all wikis |
[production] |
| 10:27 |
<jynus> |
retrying schema change on EventLogging (db1046) after failure |
[production] |
| 10:21 |
<jynus> |
End of online schema change to geo_tags; all wikis on dblist have been updated |
[production] |
| 09:34 |
<godog> |
reimage restbase-test2002.codfw.wmnet |
[production] |
| 09:23 |
<jynus> |
restarting schema change on geo_tags for all wikis |
[production] |
| 09:17 |
<mobrovac> |
restbase rolling-restart after config changes |
[production] |
| 06:55 |
<ori@tin> |
Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 18s) |
[production] |