|
2019-06-28
§
|
| 11:36 |
<elukey> |
roll restart eventstreams on all scb1* nodes |
[production] |
| 11:33 |
<elukey> |
restart eventstreams on scb1001 |
[production] |
| 11:18 |
<fsero> |
draining kubernetes1006 for applying updates |
[production] |
| 11:14 |
<fsero> |
draining kubernetes1005 for applying updates |
[production] |
| 11:13 |
<fsero> |
draining kubernetes2006 for applying updates |
[production] |
| 11:09 |
<fsero> |
draining kubernetes2005 for applying updates |
[production] |
| 11:04 |
<_joe_> |
uploading php-wmerrors to thirdparty/php72 - T187147 |
[production] |
| 10:31 |
<Reedy> |
running `foreachwiki extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --audio --mime=audio/midi --missing --throttle` on mwmaint1002 in screen T226713 |
[production] |
| 10:20 |
<reedy@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/TimedMediaHandler/maintenance/requeueTranscodes.php: Extra filtering option (duration: 00m 51s) |
[production] |
| 10:09 |
<ema> |
pool cp2008 w/ ATS backend T226637 |
[production] |
| 09:28 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:17 |
<ema> |
depool cp2008 and reimage as upload_ats T226637 |
[production] |
| 09:16 |
<elukey> |
systemctl reset-failed kafka* units on kafka2002 (role spare, failed units, already masked) |
[production] |
| 09:16 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:15 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:10 |
<moritzm> |
rebooting releases* hosts for MDS-enabled qemu/kernel |
[production] |
| 09:10 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:10 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:43 |
<elukey> |
roll restart of eventstreams on all scb2* nodes, service now working (kafka transport failures logged) |
[production] |
| 08:02 |
<moritzm> |
updating openssl packages on mw1265 |
[production] |
| 07:57 |
<ema> |
pool cp2005 w/ ATS backend T226637 |
[production] |
| 07:11 |
<_joe_> |
upgrading php-wikidiff2 on the mw canaries, only on php7 - T223391 |
[production] |
| 07:05 |
<ema> |
depool cp2005 and reimage as upload_ats T226637 |
[production] |
| 01:22 |
<Krinkle> |
Killing arclamp-log on webperf1002, no flame graphs for three days, presumably mwlog/redis connection dropped again. T215740 |
[production] |
|
2019-06-27
§
|
| 23:28 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/TimedMediaHandler/: T226748 (duration: 00m 50s) |
[production] |
| 23:26 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/GrowthExperiments/includes/HomepageHooks.php: Fix JS error on Special:Homepage (duration: 00m 50s) |
[production] |
| 23:25 |
<brion> |
roan is fixing deploy of T226748 which failed to include the patch (whoops) |
[production] |
| 21:58 |
<cdanis> |
cdanis@cp1075.eqiad.wmnet ~ % sudo -i varnish-backend-restart |
[production] |
| 21:44 |
<brion> |
deploying fix for TMH jobqueue bug T226748 |
[production] |
| 20:31 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/MobileFrontend/resources/dist: T221191: Log editor switches to visualeditorfeatureuse (duration: 00m 50s) |
[production] |
| 20:18 |
<ladsgroup@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/Wikibase: [[gerrit:519492|Avoid inserting a new addUsage job when the current usage stays untouched (duration: 01m 14s) |
[production] |
| 19:23 |
<Urbanecm> |
run namespaceDupes.php for wikis in P8674 (T173070) |
[production] |
| 19:23 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.11 refs T220736 |
[production] |
| 19:16 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855 (duration: 11m 21s) |
[production] |
| 19:04 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855 |
[production] |
| 18:52 |
<Urbanecm> |
Morning SWAT done for real |
[production] |
| 18:51 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[:gerrit:519484|Tidy up GroupOverrides]] (T173070) (duration: 00m 56s) |
[production] |
| 18:49 |
<urbanecm@deploy1001> |
Synchronized dblists/commonsuploads.dblist: [[:gerrit:Tidy up GroupOverrides]], part 1 (T173070) (duration: 00m 57s) |
[production] |
| 18:48 |
<Urbanecm> |
foreachwiki namespaceDupes.php --fix done (T173070) |
[production] |
| 18:46 |
<Urbanecm> |
Reopen Morning SWAT |
[production] |
| 18:33 |
<legoktm> |
gerrit set-account --active '"Dzahn"' |
[production] |
| 18:33 |
<Urbanecm> |
Morning SWAT done, namespaceDupes.php still running for T173070 |
[production] |
| 18:29 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[:gerrit:519180|Tidy up groupOverrides]] (T185898) (duration: 00m 56s) |
[production] |
| 18:22 |
<urbanecm@deploy1001> |
Synchronized dblists/commonsuploads.dblist: [[:gerrit:519475|Remove several wikis from commonsuploads.dblist]] (T185898) (duration: 00m 57s) |
[production] |
| 18:20 |
<urbanecm@deploy1001> |
Synchronized dblists/commonsuploads.dblist: [[:gerrit:516623|Restrict uploading on wikimaniawiki]] (T225505) (duration: 00m 56s) |
[production] |
| 18:18 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[:gerrit:516623|Restrict uploading on wikimaniawiki]], [[:gerrit:519477|Add + in front of wikimaniawiki in GroupOverrides]] (T225505) (duration: 00m 57s) |
[production] |
| 18:13 |
<herron> |
kafka2001 -> kafka-main2001 migration complete. re-enabling alerting on kafka-main2001, and moving kafka2001 to role::spare::system T225005 |
[production] |
| 18:08 |
<Urbanecm> |
running namespaceDupes.php across all wikis in tmux on mwmaint1002 (T173070) |
[production] |
| 18:06 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855, restbase1016 (duration: 01m 41s) |
[production] |