|
2020-01-15
ยง
|
| 14:54 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1242.eqiad.wmnet |
[production] |
| 14:54 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1241.eqiad.wmnet |
[production] |
| 14:54 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1240.eqiad.wmnet |
[production] |
| 14:54 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1239.eqiad.wmnet |
[production] |
| 14:54 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1238.eqiad.wmnet |
[production] |
| 14:54 |
<effie> |
lower weights on slower servers mw1238-mw1252 |
[production] |
| 14:53 |
<effie> |
pool mw1238, mw1240, mw1246 |
[production] |
| 14:43 |
<XioNoX> |
reject RPKI invalids in dfw - T220669 |
[production] |
| 14:30 |
<moritzm> |
rolling restart of FPM on mw1261-mw1265 to pick up OpenSSL security update |
[production] |
| 14:25 |
<XioNoX> |
reject RPKI invalids in ams - T220669 |
[production] |
| 14:18 |
<godog> |
reenable puppet on cp hosts, after https://gerrit.wikimedia.org/r/c/operations/puppet/+/563430 deployment |
[production] |
| 14:08 |
<effie> |
depool mw1238, mw1240, mw1246 |
[production] |
| 14:06 |
<liw@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.15 (duration: 01m 07s) |
[production] |
| 14:05 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.15 |
[production] |
| 13:58 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:56 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:54 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 13:54 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 13:53 |
<akosiaris> |
update calico policy on eqiad/codfw/staging. Add new urldownloaders. T224551 |
[production] |
| 13:52 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 13:02 |
<_joe_> |
restarting gerrit |
[production] |
| 12:50 |
<XioNoX> |
reject RPKI invalids in eqsin - T220669 |
[production] |
| 12:38 |
<vgutierrez> |
Pooling ulsfo for ncredir service - T242321 |
[production] |
| 12:27 |
<awight> |
EU SWAT done |
[production] |
| 12:24 |
<awight@deploy1001> |
Synchronized php-1.35.0-wmf.14/extensions/Cite: SWAT: [[gerrit:564002|Don't fail with a LogicException during section preview (T242434)]] (duration: 01m 10s) |
[production] |
| 12:22 |
<vgutierrez> |
upgrading ats on cp4026, cp4032, cp5006 and cp5012 - T242778 T242620 |
[production] |
| 12:06 |
<XioNoX> |
reject RPKI invalids in ulsfo - T220669 |
[production] |
| 11:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1112', diff saved to https://phabricator.wikimedia.org/P10161 and previous config saved to /var/cache/conftool/dbconfig/20200115-115826-marostegui.json |
[production] |
| 11:36 |
<elukey> |
restart all varnishkafka daemons on cp4031 |
[production] |
| 11:09 |
<legoktm> |
added SonarQubeBot to "Non-Interactive Users" group on Gerrit |
[production] |
| 10:38 |
<moritzm> |
installing openssl1.0 updates on stretch (update to 1.0.2u) |
[production] |
| 10:08 |
<ema> |
cache: rolling varnish-frontend-restart to add CAP_KILL to varnish-frontend.service T242411 |
[production] |
| 09:55 |
<vgutierrez> |
repooling cp5012 |
[production] |
| 09:46 |
<vgutierrez> |
depooling cp5012 for some ats parent select tests |
[production] |
| 09:42 |
<XioNoX> |
enable ping offload in esams - T190090 |
[production] |
| 09:32 |
<marostegui> |
Deploy schema change on x1 eqiad hosts T242749 |
[production] |
| 09:19 |
<elukey> |
roll-restart druid brokers on druid100[4-6] - locked up after segments deletion |
[production] |
| 09:11 |
<marostegui> |
Deploy schema change on x1 codfw - T242749 |
[production] |
| 08:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1098:3316 and db1098:3317', diff saved to https://phabricator.wikimedia.org/P10160 and previous config saved to /var/cache/conftool/dbconfig/20200115-085145-marostegui.json |
[production] |
| 08:44 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 08:40 |
<godog> |
roll restart ores in codfw/eqiad to apply logging pipeline changes |
[production] |
| 08:40 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 08:40 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) |
[production] |
| 08:40 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 08:23 |
<godog> |
roll restart ores in codfw/eqiad to apply logging pipeline changes |
[production] |
| 08:13 |
<godog> |
testing ores logging to pipeline on ores2001 |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1098:3316 and db1098:3317', diff saved to https://phabricator.wikimedia.org/P10159 and previous config saved to /var/cache/conftool/dbconfig/20200115-070201-marostegui.json |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1098:3316 and db1098:3317', diff saved to https://phabricator.wikimedia.org/P10158 and previous config saved to /var/cache/conftool/dbconfig/20200115-065353-marostegui.json |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1080', diff saved to https://phabricator.wikimedia.org/P10157 and previous config saved to /var/cache/conftool/dbconfig/20200115-065305-marostegui.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1080', diff saved to https://phabricator.wikimedia.org/P10156 and previous config saved to /var/cache/conftool/dbconfig/20200115-064606-marostegui.json |
[production] |