|
2015-05-15
§
|
| 17:07 |
<godog> |
still seeing metrics from xhprof creating, looking for source |
[production] |
| 16:29 |
<godog> |
bounce carbon on graphite1001 |
[production] |
| 16:23 |
<manybubbles> |
elastic1023 and elastic1024 (skipped one log) es-tool restart-fast |
[production] |
| 16:16 |
<godog> |
bounce statsdlb on graphite1001 |
[production] |
| 14:49 |
<jynus> |
migrating mariadb service from virt1000 to m5-master |
[production] |
| 14:37 |
<manybubbles> |
elastic1021 es-tool restart-fast |
[production] |
| 14:26 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1053 in s1, warm up (duration: 00m 13s) |
[production] |
| 12:21 |
<manybubbles> |
elastic1020 es-tool restart-fast |
[production] |
| 10:19 |
<godog> |
bounce statsite and uwsgi on graphite1001 |
[production] |
| 09:29 |
<godog> |
restart carbon on graphite1001 |
[production] |
| 09:15 |
<godog> |
restart hhvm on mw1018, straggling |
[production] |
| 09:07 |
<godog> |
rm MediaWiki.run_init from graphite1001 / graphite2001 |
[production] |
| 09:04 |
<ori> |
restarted hhvm / jobrunner on jobrunners to force them to pick up I6a516a0da ; re-cleared /var/lib/carbon/whisper/MediaWiki/query_* on graphite1001 and graphite2001 |
[production] |
| 08:49 |
<kart_> |
Updated cxserver to 1cb6cec |
[production] |
| 08:21 |
<jynus> |
reenabling icinga check for MySQL on db1009 |
[production] |
| 08:15 |
<oblivian> |
Synchronized wmf-config/StartProfiler.php: Null-sync to touch the file (duration: 00m 12s) |
[production] |
| 07:20 |
<ori> |
rm -rf /var/lib/carbon/whisper/MediaWiki/query_* on graphite1001 and graphite2001, as follow-up cleanup for I6a516a0da |
[production] |
| 07:14 |
<ori> |
Synchronized wmf-config/StartProfiler.php: I6a516a0da: Don't send profiling data to graphite for now (duration: 00m 11s) |
[production] |
| 06:23 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri May 15 06:22:19 UTC 2015 (duration 22m 18s) |
[production] |
| 05:38 |
<jynus> |
temporarily opening mysql port on firewall from db1009 to virt1000 |
[production] |
| 04:37 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1018, warm up (duration: 00m 11s) |
[production] |
| 02:58 |
<LocalisationUpdate> |
completed (1.26wmf6) at 2015-05-15 02:56:59+00:00 |
[production] |
| 02:55 |
<springle> |
xtrabackup clone db1057 to db1053 |
[production] |
| 02:54 |
<l10nupdate> |
Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 04m 37s) |
[production] |
| 02:42 |
<springle> |
upgrade db1053 trusty |
[production] |
| 02:34 |
<LocalisationUpdate> |
completed (1.26wmf5) at 2015-05-15 02:33:18+00:00 |
[production] |
| 02:33 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1019; depool db1053 (duration: 00m 13s) |
[production] |
| 02:30 |
<l10nupdate> |
Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 39s) |
[production] |
| 02:12 |
<manybubbles|away> |
elastic1019 es-tool restart-fast |
[production] |
| 01:13 |
<manybubbles|away> |
elastic1018 es-tool restart-fast |
[production] |
| 00:07 |
<manybubbles|away> |
elastic1017 es-tool restart-fast |
[production] |
|
2015-05-14
§
|
| 23:35 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s) |
[production] |
| 23:20 |
<ori> |
Depooled mw1169; HHVM deadlock à la T89912. Leaving it depooled to investigate. |
[production] |
| 23:05 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s) |
[production] |
| 23:05 |
<demon> |
Synchronized w/static/images/project-logos/urwikiquote.png: (no message) (duration: 00m 14s) |
[production] |
| 23:03 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 17s) |
[production] |
| 22:26 |
<ori> |
Synchronized wmf-config/StartProfiler.php: Icbf826a7: 1:1000 request profiling via xhprof (duration: 00m 12s) |
[production] |
| 22:23 |
<gwicke> |
deployed RESTBase v0.6.3 (fd942ac38ad) |
[production] |
| 22:20 |
<ori> |
Synchronized php-1.26wmf6/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: (no message) (duration: 00m 15s) |
[production] |
| 21:39 |
<manybubbles> |
I'm going to be done doing rolling restarts for a couple of hours. If someone wants to pick them up and do the next one after the cluster goes green again then be my guest. |
[production] |
| 21:35 |
<manybubbles> |
es-tool restart-fast on elastic1016 |
[production] |
| 21:27 |
<ori> |
Synchronized php-1.26wmf6/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: (no message) (duration: 00m 12s) |
[production] |
| 21:27 |
<ori> |
Synchronized php-1.26wmf5/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: (no message) (duration: 00m 12s) |
[production] |
| 21:14 |
<ori> |
Synchronized php-1.26wmf6/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: I3df6713a1: Log request times to StatsD (duration: 00m 13s) |
[production] |
| 21:14 |
<ori> |
Synchronized php-1.26wmf5/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: I3df6713a1: Log request times to StatsD (duration: 00m 15s) |
[production] |
| 21:11 |
<manybubbles> |
elastic1015 es-tool restart-fast |
[production] |
| 19:43 |
<robh> |
mass unsubcription in listadmins list, resulting in unsupressed mass unsubscribe notices to all listadmin email address (sorry about the emails!) |
[production] |
| 19:24 |
<legoktm> |
Synchronized php-1.26wmf5/skins/Nostalgia/skin.json: touch (duration: 00m 17s) |
[production] |
| 19:15 |
<legoktm> |
debugging on tin / mw1017 for nostalgiawiki issue |
[production] |
| 16:59 |
<^d> |
elasticsearch: set transient cluster.routing.allocation.node_concurrent_recoveries on prod cluster to 8 (default: 2) to speed up recoveries. |
[production] |