|
2015-06-24
§
|
| 10:31 |
<jgage> |
restarting kafka on analytics1021 |
[production] |
| 10:10 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: Switchover master es1008 -> es1009 (duration: 00m 12s) |
[production] |
| 09:24 |
<hashar> |
removing java 6 from gallium and lanthanum https://phabricator.wikimedia.org/T103491 |
[production] |
| 09:17 |
<hashar> |
apt-get upgrade on gallium and lanthanum |
[production] |
| 09:16 |
<jynus> |
performing a master failover of es1008 into es1009 |
[production] |
| 08:27 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: Repool es1004 (duration: 00m 14s) |
[production] |
| 05:46 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jun 24 05:46:32 UTC 2015 (duration 46m 31s) |
[production] |
| 05:12 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1045 (duration: 00m 13s) |
[production] |
| 05:03 |
<jgage> |
removed old logs and did 'apt-get clean' on analytics1021 to make space |
[production] |
| 03:00 |
<LocalisationUpdate> |
completed (1.26wmf11) at 2015-06-24 03:00:45+00:00 |
[production] |
| 02:54 |
<l10nupdate> |
Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 10m 34s) |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.26wmf10) at 2015-06-24 02:28:16+00:00 |
[production] |
| 02:24 |
<l10nupdate> |
Synchronized php-1.26wmf10/cache/l10n: (no message) (duration: 07m 21s) |
[production] |
| 01:39 |
<ori> |
Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi: I0e5f2d3b2 (duration: 00m 13s) |
[production] |
| 01:01 |
<gwicke> |
rolling restart of cassandra instances to rule out a single node in funky state causing elevated p99 latency |
[production] |
| 00:43 |
<ori> |
experimenting with httpd on mw1041 again |
[production] |
| 00:19 |
<gwicke> |
rolling restart of restbase instances to rule out backend connections as a source for high p99 latencies |
[production] |
| 00:14 |
<ori> |
experimenting with HHVM shutdown via /stop on the admin server on mw1041 |
[production] |
|
2015-06-23
§
|
| 23:38 |
<ori> |
Finished scap: scapping to all apaches for --restart test (duration: 07m 03s) |
[production] |
| 23:30 |
<ori> |
Started scap: scapping to all apaches for --restart test |
[production] |
| 23:24 |
<bblack> |
nginxes all updated for ssl stapling bugfix |
[production] |
| 23:24 |
<ori> |
Finished scap: scapping to scap-test dsh group for --restart test (duration: 06m 02s) |
[production] |
| 23:18 |
<ori> |
Started scap: scapping to scap-test dsh group for --restart test |
[production] |
| 23:16 |
<ori> |
scap aborted: scapping to scap-test dsh group for --restart test (duration: 00m 06s) |
[production] |
| 23:16 |
<ori> |
Started scap: scapping to scap-test dsh group for --restart test |
[production] |
| 22:14 |
<legoktm> |
Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php: RejectParserCacheValue may pass a WikiPage or Article (duration: 00m 13s) |
[production] |
| 22:07 |
<mutante> |
tmp. disabling puppet on mw1033 |
[production] |
| 21:53 |
<legoktm> |
Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi/SyntaxHighlight_GeSHi.class.php: (no message) (duration: 00m 15s) |
[production] |
| 21:50 |
<ori> |
Synchronized php-1.26wmf11/includes/parser/ParserCache.php: (no message) (duration: 00m 12s) |
[production] |
| 21:40 |
<mutante> |
starting instance planet1001 on ganeti1003 - cant get console |
[production] |
| 21:40 |
<legoktm> |
Synchronized php-1.26wmf11/includes/parser/ParserCache.php: (no message) (duration: 00m 13s) |
[production] |
| 21:36 |
<bd808> |
updated scap to 33f3002 (Ensure that the minimum batch size used by cluster_ssh is 1) |
[production] |
| 21:34 |
<ori> |
Synchronized php-1.26wmf11/extensions/SyntaxHighlight_GeSHi: 3c8bb2c493: Update SyntaxHighlight_GeSHi for cherry-pick (duration: 00m 13s) |
[production] |
| 20:32 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf11 |
[production] |
| 20:19 |
<mattflaschen> |
Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change to add Flow_test to enwiki (duration: 00m 11s) |
[production] |
| 19:59 |
<ori> |
scap failed: OSError [Errno 10] No child processes (duration: 01m 46s) |
[production] |
| 19:58 |
<ori> |
Started scap: (no message) |
[production] |
| 19:52 |
<ori> |
updated scap to master |
[production] |
| 19:11 |
<ori> |
running apache graceful-stop on mw1042 to test mod_status behavior during graceful stop |
[production] |
| 19:02 |
<twentyafterfour> |
Finished scap: New deployment branch: 1.26wmf11 try #2 (13 apaches failed) (duration: 03m 50s) |
[production] |
| 18:58 |
<twentyafterfour> |
Started scap: New deployment branch: 1.26wmf11 try #2 (13 apaches failed) |
[production] |
| 18:53 |
<twentyafterfour> |
Finished scap: New deployment branch: 1.26wmf11 (duration: 26m 37s) |
[production] |
| 18:31 |
<godog> |
start rolling-downgrade of cassandra to 2.1.3 T102015 |
[production] |
| 18:27 |
<twentyafterfour> |
Started scap: New deployment branch: 1.26wmf11 |
[production] |
| 18:13 |
<ori> |
Finished scap: (no message) (duration: 04m 34s) |
[production] |
| 18:11 |
<paravoid> |
reloading nginx on all cp* for reuseport |
[production] |
| 18:08 |
<ori> |
Started scap: (no message) |
[production] |
| 17:57 |
<ori> |
repooled scap-test servers (mw1170-mw1175 and mw1270-mw1275) |
[production] |
| 17:16 |
<ori> |
Finished scap: (no message) (duration: 01m 42s) |
[production] |
| 17:14 |
<ori> |
Started scap: (no message) |
[production] |