|
2015-10-24
§
|
| 19:34 |
<twentyafterfour> |
deployed https://gerrit.wikimedia.org/r/#/c/248638/ and restarted apache on iridium |
[production] |
| 06:00 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Oct 24 06:00:53 UTC 2015 (duration 0m 52s) |
[production] |
| 05:44 |
<ori@tin> |
Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Data/Index/FeatureIndex.php: (no message) (duration: 00m 17s) |
[production] |
| 05:11 |
<hoo> |
Set an email address for user "Ymnes", after request. Confirmed by several, including. |
[production] |
| 02:37 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-24 02:37:34+00:00 |
[production] |
| 02:33 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 23s) |
[production] |
| 00:21 |
<ebernhardson> |
started second machine (nobelium) performing copy of elasticsearch indices to codfw with 40 threads |
[production] |
|
2015-10-23
§
|
| 23:31 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki hewikisource |
[production] |
| 23:31 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki ukwiki |
[production] |
| 23:09 |
<hoo> |
Started a local rename for Stefan4 to Stefan2 on commons, per request (and attributed to) DerHexer |
[production] |
| 22:23 |
<robh> |
powercycled nobelium |
[production] |
| 21:24 |
<mutante> |
nobelium - powercycled, no console output |
[production] |
| 20:04 |
<jynus> |
rename user job had created lag on almost all enwiki dbs, things should be better now |
[production] |
| 19:08 |
<urandom> |
starting nodetool cleanup on restbase-test2001-b |
[production] |
| 19:07 |
<urandom> |
starting nodetool cleanup on restbase-test2001-a |
[production] |
| 18:16 |
<godog> |
bounce grafana-server on krypton |
[production] |
| 17:58 |
<YuviPanda> |
moved mounts around on nobelium, mounted bigger disk on /var/lib/elasticsearch |
[production] |
| 17:57 |
<godog> |
delete wikimedia-grid grafana dashboard (saved a copy first) heavy graphite queries |
[production] |
| 17:53 |
<YuviPanda> |
stopping elasticsearch on nobelium for https://phabricator.wikimedia.org/T114856 |
[production] |
| 17:13 |
<akosiaris> |
enable_notifications=0 in neon's icinga for a few mins while the storm dies down |
[production] |
| 16:47 |
<godog> |
stop puppet on graphite1001 and graphite-index cron, suspected root cause |
[production] |
| 16:26 |
<godog> |
uwsgi timing out while serving requests, bounce also carbon daemons |
[production] |
| 16:20 |
<godog> |
bounce uwsgi for graphite-web on graphite1001 |
[production] |
| 16:14 |
<urandom> |
starting rebuild of restbase-test2002-b |
[production] |
| 15:42 |
<jynus> |
copied wmf-mariadb10 (10.0.16-2) .deb from trusty to jessie on apt.wikimedia.org |
[production] |
| 15:42 |
<urandom> |
bouncing restbase-test2002-a Just To See |
[production] |
| 15:06 |
<urandom> |
running nodetool cleanup on restbase-test2003 |
[production] |
| 14:47 |
<godog> |
remove outdated cassandra metrics from graphite2001 |
[production] |
| 14:27 |
<godog> |
remove restbase-test2001 restbase-test2002 cassandra metrics |
[production] |
| 14:12 |
<mobrovac> |
restbase rolling-restarting after applying aab840f |
[production] |
| 13:55 |
<jynus> |
Rebooting and installing jessie on db2055-db2059 |
[production] |
| 13:45 |
<godog> |
remove 98percentile 999percentile meanRate stddev cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/ |
[production] |
| 13:33 |
<godog> |
remove 15MinuteRate cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/ |
[production] |
| 11:51 |
<godog> |
roll-restart cassandra-metrics-collector on restbase cluster after https://gerrit.wikimedia.org/r/248313 |
[production] |
| 11:14 |
<jynus> |
After days of no references on the logs to the renamed user_daily_contribs, deleting delete_user_daily_contribs table on all wikis |
[production] |
| 10:27 |
<jynus> |
retrying schema change on EventLogging (db1046) after failure |
[production] |
| 10:21 |
<jynus> |
End of online schema change to geo_tags; all wikis on dblist have been updated |
[production] |
| 09:34 |
<godog> |
reimage restbase-test2002.codfw.wmnet |
[production] |
| 09:23 |
<jynus> |
restarting schema change on geo_tags for all wikis |
[production] |
| 09:17 |
<mobrovac> |
restbase rolling-restart after config changes |
[production] |
| 06:55 |
<ori@tin> |
Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 18s) |
[production] |
| 06:55 |
<ori@tin> |
Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 17s) |
[production] |
| 05:53 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Oct 23 05:53:43 UTC 2015 (duration 53m 42s) |
[production] |
| 03:15 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.3/includes/profiler/TransactionProfiler.php: 5ef4a91480ea (duration: 00m 18s) |
[production] |
| 02:41 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-23 02:41:25+00:00 |
[production] |
| 02:36 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 04s) |
[production] |