|
2019-06-28
§
|
| 18:11 |
<joal@deploy1001> |
Started deploy [analytics/refinery@8d6fa30]: Late regular analytics weekly deploy - notebook1003 only again |
[production] |
| 18:09 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@8d6fa30]: Late regular analytics weekly deploy - notebook1003 only (duration: 00m 05s) |
[production] |
| 18:09 |
<joal@deploy1001> |
Started deploy [analytics/refinery@8d6fa30]: Late regular analytics weekly deploy - notebook1003 only |
[production] |
| 18:08 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@8d6fa30]: Late regular analytics weekly deploy - notebook1003 only (duration: 00m 04s) |
[production] |
| 18:08 |
<joal@deploy1001> |
Started deploy [analytics/refinery@8d6fa30]: Late regular analytics weekly deploy - notebook1003 only |
[production] |
| 18:06 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@8d6fa30]: Laste regular analytics weekly deploy (duration: 53m 35s) |
[production] |
| 17:53 |
<cdanis> |
increasing nginx proxy_buffer_size / proxy_buffers 02d7bcaa |
[production] |
| 17:36 |
<ottomata> |
restarting eventstreams on scb1001 with trace logging of X-Client-IP for T226808 |
[production] |
| 17:13 |
<joal@deploy1001> |
Started deploy [analytics/refinery@8d6fa30]: Laste regular analytics weekly deploy |
[production] |
| 16:35 |
<bblack> |
Raising varnish max_http_hdr (max allowed applayer response header count) from 64->128 in systemd config and live tuning - https://gerrit.wikimedia.org/r/519661 - T226840 |
[production] |
| 15:04 |
<eevans@deploy1001> |
scap-helm sessionstore finished |
[production] |
| 15:04 |
<eevans@deploy1001> |
scap-helm sessionstore cluster codfw completed |
[production] |
| 15:04 |
<eevans@deploy1001> |
scap-helm sessionstore upgrade production -f sessionstore-codfw-values.yaml stable/kask [namespace: sessionstore, clusters: codfw] |
[production] |
| 15:02 |
<eevans@deploy1001> |
scap-helm sessionstore finished |
[production] |
| 15:02 |
<eevans@deploy1001> |
scap-helm sessionstore cluster eqiad completed |
[production] |
| 15:02 |
<eevans@deploy1001> |
scap-helm sessionstore upgrade production -f sessionstore-eqiad-values.yaml stable/kask [namespace: sessionstore, clusters: eqiad] |
[production] |
| 14:48 |
<ema> |
pool cp2011 w/ ATS backend T226637 |
[production] |
| 14:47 |
<XioNoX> |
upload kafkatee to buster-wikimedia |
[production] |
| 14:11 |
<eevans@deploy1001> |
scap-helm sessionstore finished |
[production] |
| 14:11 |
<eevans@deploy1001> |
scap-helm sessionstore cluster staging completed |
[production] |
| 14:11 |
<eevans@deploy1001> |
scap-helm sessionstore upgrade staging -f sessionstore-staging-values.yaml stable/kask [namespace: sessionstore, clusters: staging] |
[production] |
| 14:07 |
<eevans@deploy1001> |
scap-helm sessionstore upgrade production -f sessionstore-staging-values.yaml stable/kask [namespace: sessionstore, clusters: staging] |
[production] |
| 14:06 |
<ema> |
depool cp2011 and reimage as upload_ats T226637 |
[production] |
| 11:36 |
<elukey> |
roll restart eventstreams on all scb1* nodes |
[production] |
| 11:33 |
<elukey> |
restart eventstreams on scb1001 |
[production] |
| 11:18 |
<fsero> |
draining kubernetes1006 for applying updates |
[production] |
| 11:14 |
<fsero> |
draining kubernetes1005 for applying updates |
[production] |
| 11:13 |
<fsero> |
draining kubernetes2006 for applying updates |
[production] |
| 11:09 |
<fsero> |
draining kubernetes2005 for applying updates |
[production] |
| 11:04 |
<_joe_> |
uploading php-wmerrors to thirdparty/php72 - T187147 |
[production] |
| 10:31 |
<Reedy> |
running `foreachwiki extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --audio --mime=audio/midi --missing --throttle` on mwmaint1002 in screen T226713 |
[production] |
| 10:20 |
<reedy@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/TimedMediaHandler/maintenance/requeueTranscodes.php: Extra filtering option (duration: 00m 51s) |
[production] |
| 10:09 |
<ema> |
pool cp2008 w/ ATS backend T226637 |
[production] |
| 09:28 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:17 |
<ema> |
depool cp2008 and reimage as upload_ats T226637 |
[production] |
| 09:16 |
<elukey> |
systemctl reset-failed kafka* units on kafka2002 (role spare, failed units, already masked) |
[production] |
| 09:16 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:15 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:10 |
<moritzm> |
rebooting releases* hosts for MDS-enabled qemu/kernel |
[production] |
| 09:10 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:10 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:43 |
<elukey> |
roll restart of eventstreams on all scb2* nodes, service now working (kafka transport failures logged) |
[production] |
| 08:02 |
<moritzm> |
updating openssl packages on mw1265 |
[production] |
| 07:57 |
<ema> |
pool cp2005 w/ ATS backend T226637 |
[production] |
| 07:11 |
<_joe_> |
upgrading php-wikidiff2 on the mw canaries, only on php7 - T223391 |
[production] |
| 07:05 |
<ema> |
depool cp2005 and reimage as upload_ats T226637 |
[production] |
| 01:22 |
<Krinkle> |
Killing arclamp-log on webperf1002, no flame graphs for three days, presumably mwlog/redis connection dropped again. T215740 |
[production] |