|
2015-05-16
§
|
| 13:27 |
<manybubbles> |
that was the last server in the elasticsearch rolling restart. all done. now we have new versions of the plugins. Lets try not to do that again. |
[production] |
| 13:25 |
<manybubbles> |
es-tool restart-fast on elastic1031 |
[production] |
| 09:15 |
<godog> |
bounce hhvm on mw1196 |
[production] |
| 09:10 |
<godog> |
bounce hhvm on mw1141 |
[production] |
| 07:49 |
<godog> |
restart hhvm on mw1234, still pushing xhprof metrics |
[production] |
| 06:03 |
<_joe_> |
killed nrpe on labvirt1003 - see T99341 |
[production] |
| 05:02 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat May 16 05:01:02 UTC 2015 (duration 1m 1s) |
[production] |
| 04:11 |
<andrewbogott> |
restarting sshd and generally poking around on labvirt1003 |
[production] |
| 02:47 |
<LocalisationUpdate> |
completed (1.26wmf6) at 2015-05-16 02:46:08+00:00 |
[production] |
| 02:43 |
<l10nupdate> |
Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 04m 55s) |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.26wmf5) at 2015-05-16 02:28:37+00:00 |
[production] |
| 02:25 |
<l10nupdate> |
Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 55s) |
[production] |
|
2015-05-15
§
|
| 22:35 |
<ejegg> |
updated crm from 03eb4cff1b009e8abaceec250f9a1c5d1f3c6b18 to 7ffe0cefb019828a09c9369187f14518847b5f41 |
[production] |
| 19:44 |
<manybubbles> |
elastic1027 es-tool restart-fast |
[production] |
| 19:37 |
<awight> |
update crm from 2a2336655737a2cd1d3cc24624d1e8475e4cf039 to 03eb4cff1b009e8abaceec250f9a1c5d1f3c6b18 |
[production] |
| 18:29 |
<manybubbles> |
elastic1026 es-tool restart-fast |
[production] |
| 18:28 |
<godog> |
bounce hhvm on mw1118 |
[production] |
| 17:55 |
<jynus> |
migrating of db service from virt1000 to m5-master aborted, service continues on virt1000 |
[production] |
| 17:44 |
<manybubbles> |
rolling restart almost done on elastic1025 - 1026 is next! |
[production] |
| 17:33 |
<andrewbogott> |
updating qemu binaries on labvirt1001 |
[production] |
| 17:29 |
<godog> |
clean up remaining xhprof metrics from graphite1001 |
[production] |
| 17:20 |
<godog> |
bounce hhvm on mw1017 |
[production] |
| 17:07 |
<godog> |
still seeing metrics from xhprof creating, looking for source |
[production] |
| 16:29 |
<godog> |
bounce carbon on graphite1001 |
[production] |
| 16:23 |
<manybubbles> |
elastic1023 and elastic1024 (skipped one log) es-tool restart-fast |
[production] |
| 16:16 |
<godog> |
bounce statsdlb on graphite1001 |
[production] |
| 14:49 |
<jynus> |
migrating mariadb service from virt1000 to m5-master |
[production] |
| 14:37 |
<manybubbles> |
elastic1021 es-tool restart-fast |
[production] |
| 14:26 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1053 in s1, warm up (duration: 00m 13s) |
[production] |
| 12:21 |
<manybubbles> |
elastic1020 es-tool restart-fast |
[production] |
| 10:19 |
<godog> |
bounce statsite and uwsgi on graphite1001 |
[production] |
| 09:29 |
<godog> |
restart carbon on graphite1001 |
[production] |
| 09:15 |
<godog> |
restart hhvm on mw1018, straggling |
[production] |
| 09:07 |
<godog> |
rm MediaWiki.run_init from graphite1001 / graphite2001 |
[production] |
| 09:04 |
<ori> |
restarted hhvm / jobrunner on jobrunners to force them to pick up I6a516a0da ; re-cleared /var/lib/carbon/whisper/MediaWiki/query_* on graphite1001 and graphite2001 |
[production] |
| 08:49 |
<kart_> |
Updated cxserver to 1cb6cec |
[production] |
| 08:21 |
<jynus> |
reenabling icinga check for MySQL on db1009 |
[production] |
| 08:15 |
<oblivian> |
Synchronized wmf-config/StartProfiler.php: Null-sync to touch the file (duration: 00m 12s) |
[production] |
| 07:20 |
<ori> |
rm -rf /var/lib/carbon/whisper/MediaWiki/query_* on graphite1001 and graphite2001, as follow-up cleanup for I6a516a0da |
[production] |
| 07:14 |
<ori> |
Synchronized wmf-config/StartProfiler.php: I6a516a0da: Don't send profiling data to graphite for now (duration: 00m 11s) |
[production] |
| 06:23 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri May 15 06:22:19 UTC 2015 (duration 22m 18s) |
[production] |
| 05:38 |
<jynus> |
temporarily opening mysql port on firewall from db1009 to virt1000 |
[production] |
| 04:37 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1018, warm up (duration: 00m 11s) |
[production] |
| 02:58 |
<LocalisationUpdate> |
completed (1.26wmf6) at 2015-05-15 02:56:59+00:00 |
[production] |
| 02:55 |
<springle> |
xtrabackup clone db1057 to db1053 |
[production] |
| 02:54 |
<l10nupdate> |
Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 04m 37s) |
[production] |
| 02:42 |
<springle> |
upgrade db1053 trusty |
[production] |
| 02:34 |
<LocalisationUpdate> |
completed (1.26wmf5) at 2015-05-15 02:33:18+00:00 |
[production] |
| 02:33 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1019; depool db1053 (duration: 00m 13s) |
[production] |
| 02:30 |
<l10nupdate> |
Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 39s) |
[production] |