|
2018-07-24
§
|
| 08:21 |
<elukey> |
rolling restart of kafka jumbo/main-(eqiad|codfw) clusters to pick up the new max open files limit (infinity -> 128k) |
[production] |
| 06:38 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1081 (duration: 00m 55s) |
[production] |
| 06:32 |
<marostegui> |
Deploy schema change on dbstore1002:s4 T144010 T51190 T199368 |
[production] |
| 05:33 |
<kartik@deploy1001> |
Finished deploy [cxserver/deploy@d378d27]: Update cxserver to d3c9d15 (T198941) (duration: 04m 01s) |
[production] |
| 05:29 |
<kartik@deploy1001> |
Started deploy [cxserver/deploy@d378d27]: Update cxserver to d3c9d15 (T198941) |
[production] |
| 05:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 00m 54s) |
[production] |
| 05:05 |
<marostegui> |
Deploy schema change on db1081 T144010 T51190 T199368 |
[production] |
| 04:54 |
<marostegui> |
Stop replication in sync on db1081 and db1121 |
[production] |
| 04:54 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1084, db1121 (duration: 00m 56s) |
[production] |
| 04:44 |
<marostegui> |
Deploy schema change on db1066 (s2 primary master) T144010 T51190 T199368 |
[production] |
| 03:03 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Tue Jul 24 03:03:08 UTC 2018 (duration 10m 22s) |
[production] |
| 02:52 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.13) (duration: 09m 19s) |
[production] |
| 02:23 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.12) (duration: 09m 44s) |
[production] |
|
2018-07-23
§
|
| 22:53 |
<mobrovac@deploy1001> |
Started restart [eventstreams/deploy@5ed03e0]: A ggodnight restart - T199813 |
[production] |
| 22:27 |
<thcipriani@deploy1001> |
Finished scap: [[gerrit:447506|Revert "Accept BCP 47 codes as aliases for nonstandard variants"]] T199941 [[gerrit:447507|Revert "Ensure LanguageCode::bcp47() returns a valid BCP 47 language code"]] T199941 (duration: 33m 21s) |
[production] |
| 21:53 |
<thcipriani@deploy1001> |
Started scap: [[gerrit:447506|Revert "Accept BCP 47 codes as aliases for nonstandard variants"]] T199941 [[gerrit:447507|Revert "Ensure LanguageCode::bcp47() returns a valid BCP 47 language code"]] T199941 |
[production] |
| 20:51 |
<ebernhar|lunch> |
T156137: restarting elastic1035 to disable G1GC |
[production] |
| 20:45 |
<ebernhar|lunch> |
T156137: restarting elastic1027 to disable G1GC |
[production] |
| 20:27 |
<arlolra> |
Updated Parsoid to a1e851c (T199808, T194083) |
[production] |
| 20:22 |
<arlolra@deploy1001> |
Finished deploy [parsoid/deploy@80384a5]: Updating Parsoid to a1e851c (duration: 09m 45s) |
[production] |
| 20:13 |
<arlolra@deploy1001> |
Started deploy [parsoid/deploy@80384a5]: Updating Parsoid to a1e851c |
[production] |
| 20:09 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@565b41a]: Update mobileapps to 254cef5 (duration: 05m 56s) |
[production] |
| 20:03 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@565b41a]: Update mobileapps to 254cef5 |
[production] |
| 19:49 |
<gehel@deploy1001> |
Finished deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (duration: 11m 35s) |
[production] |
| 19:38 |
<gehel@deploy1001> |
Started deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI |
[production] |
| 19:37 |
<gehel@deploy1001> |
Finished deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 22s) |
[production] |
| 19:36 |
<gehel@deploy1001> |
Started deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (wdqs1009 only) |
[production] |
| 17:56 |
<gehel@deploy1001> |
Finished deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 25s) |
[production] |
| 17:56 |
<gehel@deploy1001> |
Started deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) |
[production] |
| 17:53 |
<ejegg> |
updated payments-wiki from d8d8293b87 to 6390e50abd |
[production] |
| 17:47 |
<gehel@deploy1001> |
Finished deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 03s) |
[production] |
| 17:47 |
<gehel@deploy1001> |
Started deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) |
[production] |
| 17:29 |
<XenoRyet> |
updated payments-wiki from 63372154d0 to d8d8293b87 |
[production] |
| 17:16 |
<gehel> |
canceling deployment of new WDQS GUI to all servers, looks like there is a JS issue. Will reschedule once the issue is fixed. |
[production] |
| 17:07 |
<gehel@deploy1001> |
Finished deploy [wdqs/wdqs@d7e8292]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s) |
[production] |
| 15:25 |
<gehel> |
decrease gc_grace_seconds to 4 days on cassandra / maps |
[production] |
| 15:02 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@5cfdf26]: Increase concurrency back to normal levels (from 20 to 50) (duration: 01m 38s) |
[production] |
| 15:00 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@5cfdf26]: Increase concurrency back to normal levels (from 20 to 50) |
[production] |
| 14:57 |
<mobrovac@deploy1001> |
Started restart [eventstreams/deploy@5ed03e0]: Fresh start of the service after alleviating 404s |
[production] |
| 14:53 |
<elukey> |
delete empty/not-used/wrongly-created topics in Kafka main-eqiad - T199510 |
[production] |
| 14:49 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1081, db1097:3314 (duration: 00m 53s) |
[production] |
| 14:17 |
<marostegui> |
Stop replication in sync on db1081 and db2051 |
[production] |
| 14:00 |
<mobrovac@deploy1001> |
Finished deploy [eventstreams/deploy@5ed03e0]: Update node-rdkafka to v2.3.4 - T199813 (duration: 02m 29s) |
[production] |
| 13:57 |
<mobrovac@deploy1001> |
Started deploy [eventstreams/deploy@5ed03e0]: Update node-rdkafka to v2.3.4 - T199813 |
[production] |
| 13:53 |
<marostegui> |
Stop replication in sync on db1081 and db1097:3317 |
[production] |
| 13:51 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1081, db1097:3314 (duration: 00m 54s) |
[production] |
| 13:44 |
<ema> |
!log trafficserver 7.1.3+ds-4wm1 uploaded to stretch-wikimedia T200178 |
[production] |
| 13:36 |
<marostegui> |
Deploy schema change on s4 codfw master (db2051), this will generate lag on s4 codfw T144010 T51190 T199368 |
[production] |
| 13:23 |
<marostegui> |
Deploy schema change on labswiki and labstestwiki T144010 T51190 T199368 |
[production] |
| 13:13 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1074, db1105:3312 (duration: 00m 55s) |
[production] |