|
2015-06-30
§
|
| 19:30 |
<thcipriani> |
Started scap: testwiki to php-1.26wmf12 and rebuild l10n cache |
[production] |
| 19:00 |
<demon> |
Synchronized php-1.26wmf11/includes/WebResponse.php: rv my test (duration: 00m 12s) |
[production] |
| 18:55 |
<demon> |
Synchronized php-1.26wmf11/includes/WebResponse.php: (no message) (duration: 00m 12s) |
[production] |
| 18:36 |
<cmjohnson1> |
labcontrol1002 going down for a few minutes |
[production] |
| 18:33 |
<mutante> |
tendril - short downtime for switch to new repo |
[production] |
| 18:17 |
<gwicke> |
restarted cassandra on restbase1005 with g1gc GC and larger heap |
[production] |
| 18:16 |
<gwicke> |
restarted cassandra on restbase1004 with g1gc GC and larger heap |
[production] |
| 17:02 |
<akosiaris> |
enabled and ran puppet on lvs400X, lvs300X, lvs100[123]. noops |
[production] |
| 16:58 |
<bblack> |
re-enabling puppet on caches |
[production] |
| 16:52 |
<bblack> |
disabling puppet on cache clusters |
[production] |
| 16:48 |
<akosiaris> |
enabled an ran puppet on all lvs servers @ codfw |
[production] |
| 16:22 |
<akosiaris> |
enabled and ran puppet on lvs1004. noop as well |
[production] |
| 16:19 |
<akosiaris> |
enabled and running puppet on lvs1005 |
[production] |
| 16:11 |
<akosiaris> |
enabling and running puppet on lvs1006 |
[production] |
| 16:09 |
<akosiaris> |
disabling puppet on all lvs and neon |
[production] |
| 16:08 |
<gwicke> |
restarting cassandra instance on restbase1004 |
[production] |
| 15:12 |
<thcipriani> |
Synchronized wmf-config: SWAT: Standardise a ton of ticket comments [[gerrit:221803]] (duration: 00m 13s) |
[production] |
| 15:04 |
<thcipriani> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable CX all wikipedias except enwiki [[gerrit:221831]] (duration: 00m 13s) |
[production] |
| 14:46 |
<kart_> |
Update cxserver to 0d21a80 |
[production] |
| 14:10 |
<mobrovac> |
restbase restarting cassandra on restbase1005 |
[production] |
| 11:29 |
<mobrovac> |
restbase restarting cassandra on restbase1005 |
[production] |
| 10:41 |
<mobrovac> |
restbase restarting on all nodes |
[production] |
| 09:54 |
<mobrovac> |
restbase restarting cassandra on restbase1004 |
[production] |
| 08:53 |
<mobrovac> |
restbase restrting cassandra on restbase1004 |
[production] |
| 08:05 |
<jynus> |
applying schema changes for Gather extension |
[production] |
| 06:56 |
<jynus> |
initiating query profiling on db1018 |
[production] |
| 05:21 |
<gwicke> |
restarting cassandra instance on restbase1004; was in small-write mode |
[production] |
| 05:17 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1034 (duration: 00m 12s) |
[production] |
| 04:37 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Jun 30 04:37:00 UTC 2015 (duration 36m 59s) |
[production] |
| 02:22 |
<LocalisationUpdate> |
completed (1.26wmf11) at 2015-06-30 02:22:00+00:00 |
[production] |
| 02:18 |
<l10nupdate> |
Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 06m 09s) |
[production] |
| 02:11 |
<krenair> |
Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 12s) |
[production] |
| 01:56 |
<krenair> |
Synchronized wmf-config/wikitech.php: (no message) (duration: 00m 11s) |
[production] |
| 01:41 |
<krinkle> |
Synchronized php-1.26wmf11/includes/resourceloader/ResourceLoader.php: I7761242f01 (duration: 00m 14s) |
[production] |
| 00:37 |
<godog> |
restbase1* upgrade to cassandra 2.1.7 completed |
[production] |
|
2015-06-29
§
|
| 23:57 |
<robh> |
mw2027 was offline (blank screen on serial console). mgmt powercycled |
[production] |
| 23:48 |
<godog> |
start upgrading restbase1* to cassandra 2.1.7 |
[production] |
| 23:41 |
<gwicke> |
restarted cassandra instance on restbase1004.eqiad; log showed many small writes and clients saw timeouts |
[production] |
| 23:29 |
<gwicke> |
deployed restbase 32db4ce1e1 |
[production] |
| 23:21 |
<ori> |
Synchronized php-1.26wmf11/includes/resourceloader: I0e5f2d3b2: resourceloader: Add timing metrics for key operations (duration: 01m 12s) |
[production] |
| 23:15 |
<catrope> |
Synchronized wmf-config/: wikitech cleanup (duration: 01m 08s) |
[production] |
| 23:11 |
<RoanKattouw> |
ssh: connect to host mw2027.codfw.wmnet port 22: Connection timed out |
[production] |
| 23:11 |
<RoanKattouw> |
Synced wmf-config/CommonSettings.php: Remove survey access point in Popups |
[production] |
| 23:09 |
<godog> |
stop ircecho on neon, icinga spam |
[production] |
| 22:53 |
<gwicke> |
canary deploy of restbase 32db4ce1e1 on restbase1001.eqiad |
[production] |
| 21:30 |
<urandom> |
restarting restbase1004 to apply new metrics reporting interval |
[production] |
| 20:19 |
<subbu> |
deployed parsoid sha ea98be88 |
[production] |
| 18:18 |
<ori> |
Synchronized php-1.26wmf11/includes/db/LoadBalancer.php: I0e5f2d3b2: Use APC for caching slave lag times (duration: 01m 09s) |
[production] |
| 18:00 |
<cmjohnson1> |
powering down ms-be1015 |
[production] |
| 16:06 |
<bblack> |
re-enabling puppet on caches |
[production] |