|
2015-10-23
§
|
| 20:04 |
<jynus> |
rename user job had created lag on almost all enwiki dbs, things should be better now |
[production] |
| 19:08 |
<urandom> |
starting nodetool cleanup on restbase-test2001-b |
[production] |
| 19:07 |
<urandom> |
starting nodetool cleanup on restbase-test2001-a |
[production] |
| 18:16 |
<godog> |
bounce grafana-server on krypton |
[production] |
| 17:58 |
<YuviPanda> |
moved mounts around on nobelium, mounted bigger disk on /var/lib/elasticsearch |
[production] |
| 17:57 |
<godog> |
delete wikimedia-grid grafana dashboard (saved a copy first) heavy graphite queries |
[production] |
| 17:53 |
<YuviPanda> |
stopping elasticsearch on nobelium for https://phabricator.wikimedia.org/T114856 |
[production] |
| 17:13 |
<akosiaris> |
enable_notifications=0 in neon's icinga for a few mins while the storm dies down |
[production] |
| 16:47 |
<godog> |
stop puppet on graphite1001 and graphite-index cron, suspected root cause |
[production] |
| 16:26 |
<godog> |
uwsgi timing out while serving requests, bounce also carbon daemons |
[production] |
| 16:20 |
<godog> |
bounce uwsgi for graphite-web on graphite1001 |
[production] |
| 16:14 |
<urandom> |
starting rebuild of restbase-test2002-b |
[production] |
| 15:42 |
<jynus> |
copied wmf-mariadb10 (10.0.16-2) .deb from trusty to jessie on apt.wikimedia.org |
[production] |
| 15:42 |
<urandom> |
bouncing restbase-test2002-a Just To See |
[production] |
| 15:06 |
<urandom> |
running nodetool cleanup on restbase-test2003 |
[production] |
| 14:47 |
<godog> |
remove outdated cassandra metrics from graphite2001 |
[production] |
| 14:27 |
<godog> |
remove restbase-test2001 restbase-test2002 cassandra metrics |
[production] |
| 14:12 |
<mobrovac> |
restbase rolling-restarting after applying aab840f |
[production] |
| 13:55 |
<jynus> |
Rebooting and installing jessie on db2055-db2059 |
[production] |
| 13:45 |
<godog> |
remove 98percentile 999percentile meanRate stddev cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/ |
[production] |
| 13:33 |
<godog> |
remove 15MinuteRate cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/ |
[production] |
| 11:51 |
<godog> |
roll-restart cassandra-metrics-collector on restbase cluster after https://gerrit.wikimedia.org/r/248313 |
[production] |
| 11:14 |
<jynus> |
After days of no references on the logs to the renamed user_daily_contribs, deleting delete_user_daily_contribs table on all wikis |
[production] |
| 10:27 |
<jynus> |
retrying schema change on EventLogging (db1046) after failure |
[production] |
| 10:21 |
<jynus> |
End of online schema change to geo_tags; all wikis on dblist have been updated |
[production] |
| 09:34 |
<godog> |
reimage restbase-test2002.codfw.wmnet |
[production] |
| 09:23 |
<jynus> |
restarting schema change on geo_tags for all wikis |
[production] |
| 09:17 |
<mobrovac> |
restbase rolling-restart after config changes |
[production] |
| 06:55 |
<ori@tin> |
Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 18s) |
[production] |
| 06:55 |
<ori@tin> |
Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 17s) |
[production] |
| 05:53 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Oct 23 05:53:43 UTC 2015 (duration 53m 42s) |
[production] |
| 03:15 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.3/includes/profiler/TransactionProfiler.php: 5ef4a91480ea (duration: 00m 18s) |
[production] |
| 02:41 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-23 02:41:25+00:00 |
[production] |
| 02:36 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 04s) |
[production] |
| 00:10 |
<krenair@tin> |
Synchronized w/static/images/project-logos/anwiki.png: https://gerrit.wikimedia.org/r/#/c/247253/ (duration: 00m 16s) |
[production] |
|
2015-10-22
§
|
| 23:58 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247850/ (duration: 00m 17s) |
[production] |
| 23:49 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWMediaDialog.js: https://gerrit.wikimedia.org/r/#/c/248273/ (duration: 00m 18s) |
[production] |
| 23:13 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.3/extensions/Flow/container.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s) |
[production] |
| 23:13 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.3/extensions/Flow/autoload.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s) |
[production] |
| 23:12 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/SpamFilter/RateLimits.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s) |
[production] |
| 22:55 |
<ori@tin> |
Synchronized php-1.27.0-wmf.3/extensions/Cite: extensions/Cite update which fell off the SWAT train yesterday (duration: 00m 19s) |
[production] |
| 22:02 |
<ori@tin> |
Synchronized php-1.27.0-wmf.3/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s) |
[production] |
| 22:02 |
<ori@tin> |
Synchronized php-1.27.0-wmf.2/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s) |
[production] |
| 21:11 |
<reedy@tin> |
Synchronized docroot and w: Add more dblist symlinks (duration: 00m 18s) |
[production] |
| 21:10 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.3/includes/changetags/ChangeTags.php: e7126ed331109 (duration: 00m 17s) |
[production] |
| 21:05 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.3/includes: a6262272c9666d (duration: 00m 23s) |
[production] |
| 20:39 |
<yurik@tin> |
Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner/: Deploying ZeroBanner T116309 patch 248239 (duration: 00m 18s) |
[production] |
| 20:38 |
<yurik@tin> |
Synchronized php-1.27.0-wmf.3/extensions/MobileFrontend: Deploying MobileFrontend T116309 patch 248238 (duration: 00m 36s) |
[production] |
| 20:26 |
<bd808> |
Removed "zirconium.wikimedia.org" from Trebuchet's minion list for iegreview/iegreview |
[production] |
| 20:12 |
<bd808> |
Updated iegreview.wikimedia.org to bcaf23b (Fix logger usage in Controllers\Account\Recover) |
[production] |