|
2019-06-28
§
|
| 17:36 |
<ottomata> |
restarting eventstreams on scb1001 with trace logging of X-Client-IP for T226808 |
[production] |
| 17:13 |
<joal@deploy1001> |
Started deploy [analytics/refinery@8d6fa30]: Laste regular analytics weekly deploy |
[production] |
| 16:35 |
<bblack> |
Raising varnish max_http_hdr (max allowed applayer response header count) from 64->128 in systemd config and live tuning - https://gerrit.wikimedia.org/r/519661 - T226840 |
[production] |
| 15:04 |
<eevans@deploy1001> |
scap-helm sessionstore finished |
[production] |
| 15:04 |
<eevans@deploy1001> |
scap-helm sessionstore cluster codfw completed |
[production] |
| 15:04 |
<eevans@deploy1001> |
scap-helm sessionstore upgrade production -f sessionstore-codfw-values.yaml stable/kask [namespace: sessionstore, clusters: codfw] |
[production] |
| 15:02 |
<eevans@deploy1001> |
scap-helm sessionstore finished |
[production] |
| 15:02 |
<eevans@deploy1001> |
scap-helm sessionstore cluster eqiad completed |
[production] |
| 15:02 |
<eevans@deploy1001> |
scap-helm sessionstore upgrade production -f sessionstore-eqiad-values.yaml stable/kask [namespace: sessionstore, clusters: eqiad] |
[production] |
| 14:48 |
<ema> |
pool cp2011 w/ ATS backend T226637 |
[production] |
| 14:47 |
<XioNoX> |
upload kafkatee to buster-wikimedia |
[production] |
| 14:11 |
<eevans@deploy1001> |
scap-helm sessionstore finished |
[production] |
| 14:11 |
<eevans@deploy1001> |
scap-helm sessionstore cluster staging completed |
[production] |
| 14:11 |
<eevans@deploy1001> |
scap-helm sessionstore upgrade staging -f sessionstore-staging-values.yaml stable/kask [namespace: sessionstore, clusters: staging] |
[production] |
| 14:07 |
<eevans@deploy1001> |
scap-helm sessionstore upgrade production -f sessionstore-staging-values.yaml stable/kask [namespace: sessionstore, clusters: staging] |
[production] |
| 14:06 |
<ema> |
depool cp2011 and reimage as upload_ats T226637 |
[production] |
| 11:36 |
<elukey> |
roll restart eventstreams on all scb1* nodes |
[production] |
| 11:33 |
<elukey> |
restart eventstreams on scb1001 |
[production] |
| 11:18 |
<fsero> |
draining kubernetes1006 for applying updates |
[production] |
| 11:14 |
<fsero> |
draining kubernetes1005 for applying updates |
[production] |
| 11:13 |
<fsero> |
draining kubernetes2006 for applying updates |
[production] |
| 11:09 |
<fsero> |
draining kubernetes2005 for applying updates |
[production] |
| 11:04 |
<_joe_> |
uploading php-wmerrors to thirdparty/php72 - T187147 |
[production] |
| 10:31 |
<Reedy> |
running `foreachwiki extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --audio --mime=audio/midi --missing --throttle` on mwmaint1002 in screen T226713 |
[production] |
| 10:20 |
<reedy@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/TimedMediaHandler/maintenance/requeueTranscodes.php: Extra filtering option (duration: 00m 51s) |
[production] |
| 10:09 |
<ema> |
pool cp2008 w/ ATS backend T226637 |
[production] |
| 09:28 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:17 |
<ema> |
depool cp2008 and reimage as upload_ats T226637 |
[production] |
| 09:16 |
<elukey> |
systemctl reset-failed kafka* units on kafka2002 (role spare, failed units, already masked) |
[production] |
| 09:16 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:15 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:10 |
<moritzm> |
rebooting releases* hosts for MDS-enabled qemu/kernel |
[production] |
| 09:10 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:10 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:43 |
<elukey> |
roll restart of eventstreams on all scb2* nodes, service now working (kafka transport failures logged) |
[production] |
| 08:02 |
<moritzm> |
updating openssl packages on mw1265 |
[production] |
| 07:57 |
<ema> |
pool cp2005 w/ ATS backend T226637 |
[production] |
| 07:11 |
<_joe_> |
upgrading php-wikidiff2 on the mw canaries, only on php7 - T223391 |
[production] |
| 07:05 |
<ema> |
depool cp2005 and reimage as upload_ats T226637 |
[production] |
| 01:22 |
<Krinkle> |
Killing arclamp-log on webperf1002, no flame graphs for three days, presumably mwlog/redis connection dropped again. T215740 |
[production] |
|
2019-06-27
§
|
| 23:28 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/TimedMediaHandler/: T226748 (duration: 00m 50s) |
[production] |
| 23:26 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/GrowthExperiments/includes/HomepageHooks.php: Fix JS error on Special:Homepage (duration: 00m 50s) |
[production] |
| 23:25 |
<brion> |
roan is fixing deploy of T226748 which failed to include the patch (whoops) |
[production] |
| 21:58 |
<cdanis> |
cdanis@cp1075.eqiad.wmnet ~ % sudo -i varnish-backend-restart |
[production] |
| 21:44 |
<brion> |
deploying fix for TMH jobqueue bug T226748 |
[production] |
| 20:31 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/MobileFrontend/resources/dist: T221191: Log editor switches to visualeditorfeatureuse (duration: 00m 50s) |
[production] |
| 20:18 |
<ladsgroup@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/Wikibase: [[gerrit:519492|Avoid inserting a new addUsage job when the current usage stays untouched (duration: 01m 14s) |
[production] |
| 19:23 |
<Urbanecm> |
run namespaceDupes.php for wikis in P8674 (T173070) |
[production] |
| 19:23 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.11 refs T220736 |
[production] |