|
2020-02-11
§
|
| 10:18 |
<mvolz@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:11 |
<mvolz@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'citoid' for release 'staging' . |
[production] |
| 10:07 |
<vgutierrez> |
rolling restart of ats-tls in ulsfo - T244464 |
[production] |
| 09:57 |
<vgutierrez> |
depool cp3063 and cp3064 and reimage as buster - T242093 |
[production] |
| 09:52 |
<vgutierrez> |
depool cp5006 and reimage as buster - T242093 |
[production] |
| 09:52 |
<vgutierrez> |
pool cp5007 running buster - T242093 |
[production] |
| 08:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase db1107 weight from 10 to 11', diff saved to https://phabricator.wikimedia.org/P10380 and previous config saved to /var/cache/conftool/dbconfig/20200211-083812-marostegui.json |
[production] |
| 08:25 |
<marostegui> |
Upgrade db1095:3312, db1095:3313 |
[production] |
| 08:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1013 after upgrade', diff saved to https://phabricator.wikimedia.org/P10379 and previous config saved to /var/cache/conftool/dbconfig/20200211-082204-marostegui.json |
[production] |
| 08:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1013 after upgrade', diff saved to https://phabricator.wikimedia.org/P10378 and previous config saved to /var/cache/conftool/dbconfig/20200211-081421-marostegui.json |
[production] |
| 08:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 5 to 10 for db1107 - T242702', diff saved to https://phabricator.wikimedia.org/P10377 and previous config saved to /var/cache/conftool/dbconfig/20200211-081319-marostegui.json |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1013 after upgrade', diff saved to https://phabricator.wikimedia.org/P10376 and previous config saved to /var/cache/conftool/dbconfig/20200211-080458-marostegui.json |
[production] |
| 07:57 |
<akosiaris> |
T242705 systemctl stop uwsgi-ores on ores2001. |
[production] |
| 07:54 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 07:54 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1013 after upgrade', diff saved to https://phabricator.wikimedia.org/P10375 and previous config saved to /var/cache/conftool/dbconfig/20200211-075358-marostegui.json |
[production] |
| 07:47 |
<marostegui> |
Upgrade es1013 - T239791 |
[production] |
| 07:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1013 - T239791', diff saved to https://phabricator.wikimedia.org/P10374 and previous config saved to /var/cache/conftool/dbconfig/20200211-074358-marostegui.json |
[production] |
| 07:23 |
<vgutierrez> |
depool cp5007 and reimage as buster - T242093 |
[production] |
| 07:22 |
<vgutierrez> |
pool cp5001 and cp5008 running buster - T242093 |
[production] |
| 07:21 |
<marostegui> |
Remove partitions from db2086:3318 - T239453 |
[production] |
| 07:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2086:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10373 and previous config saved to /var/cache/conftool/dbconfig/20200211-071936-marostegui.json |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2085:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10372 and previous config saved to /var/cache/conftool/dbconfig/20200211-071639-marostegui.json |
[production] |
| 07:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10371 and previous config saved to /var/cache/conftool/dbconfig/20200211-070720-marostegui.json |
[production] |
| 07:01 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:59 |
<marostegui> |
Stop haproxy on dbproxy1001 - T244463 |
[production] |
| 06:59 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:58 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:57 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:48 |
<marostegui> |
Remove grants in m1 for dbproxy1001 - T231280 |
[production] |
| 06:25 |
<vgutierrez> |
depool cp5001 & cp5008 and reimage as buster - T242093 |
[production] |
| 06:18 |
<marostegui> |
Failover m1-master from dbproxy1014 to dbproxy1012 - T202367 |
[production] |
| 00:26 |
<ebernhardson@deploy1001> |
Synchronized php-1.35.0-wmf.18/skins/MinervaNeue: SWAT: Revert: Reduce userContributions icon code (duration: 01m 06s) |
[production] |
| 00:20 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Give NS_HELP same weight as NS_MAIN in search on wikitech (duration: 01m 06s) |
[production] |
| 00:15 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/: SWAT: Enable SpecialMute page on all wikis (duration: 01m 06s) |
[production] |
|
2020-02-10
§
|
| 23:30 |
<robh> |
cp108[23] returned to service via T243167 |
[production] |
| 23:28 |
<legoktm> |
restarting zuul |
[production] |
| 23:26 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.18/extensions/OATHAuth/src/Key/TOTPKey.php: T244308 (duration: 01m 04s) |
[production] |
| 23:25 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.16/extensions/OATHAuth/src/Key/TOTPKey.php: T244308 (duration: 01m 07s) |
[production] |
| 23:06 |
<robh> |
cp108[01] returned to service, cp108[23] offline for bios update via T243167 |
[production] |
| 22:50 |
<chasemp> |
phab1001:~# sudo /srv/phab/phabricator/bin/bulk make-silent --id 2164 |
[production] |
| 22:45 |
<sbassett@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add authevents as monolog channel (duration: 01m 06s) |
[production] |
| 22:43 |
<robh> |
cp107[789] returned to service, cp108[01] offline for bios update via T243167 |
[production] |
| 22:42 |
<robh> |
cp107[89] returned to service, cp108[01] offline for bios update via T243167 |
[production] |
| 21:58 |
<robh> |
cp107[56] returned to service, cp107[78] offline for bios update via T243167 |
[production] |
| 21:43 |
<arlolra> |
Updated Parsoid to 612106d2 (T244412, T244413, T242746, T235273, T235307, T238845, T204618, T240054) |
[production] |
| 21:38 |
<robh> |
cp1075 & cp1076 offline for bios updates per T243167 |
[production] |
| 21:36 |
<robh> |
cp1075 and cp1076 going offline for bios updates. This will cause a bit of cp irc icinga noise, but no paging. Not putting into maint mode, as there is no way to maint mode the noisest check (which checks all backends and thus shouldnt be disabled) |
[production] |
| 21:33 |
<arlolra@deploy1001> |
Finished deploy [parsoid/deploy@d2d4870]: Updating Parsoid to 612106d2 (duration: 10m 26s) |
[production] |
| 21:32 |
<XioNoX> |
clamp tcp-mss on cr2-eqiad:xe-3/3/3 |
[production] |