|
2020-02-07
ยง
|
| 16:01 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@bc777d6]: Fix for T243422 |
[production] |
| 15:50 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:48 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:25 |
<vgutierrez> |
depool & reimage cp4030 as buster - T242093 |
[production] |
| 15:21 |
<vgutierrez> |
pooling cp4031 with buster - T242093 |
[production] |
| 15:20 |
<vgutierrez> |
pooling ncredir3001 running buster - T243391 |
[production] |
| 15:18 |
<marostegui> |
Restart all instances on db1124 and db1125 to pick up a new replication filter - T240094 |
[production] |
| 15:11 |
<marostegui> |
Restart all instances on db2094 and db2095 to pick up a new replication filter - T240094 |
[production] |
| 14:56 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:53 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:43 |
<hoo@deploy1001> |
Synchronized wmf-config/Wikibase.php: REVERT: Wikibase Client: Fix setting name typo (T244529) (duration: 01m 40s) |
[production] |
| 14:43 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript createAndPromote.php --wiki=zhwiki --force "Amir Sarabadani (WMDE)" --sysop (T244578) |
[production] |
| 14:40 |
<hoo@deploy1001> |
Scap failed!: 9/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 14:38 |
<hoo@deploy1001> |
Synchronized wmf-config/Wikibase.php: Wikibase Client: Fix setting name typo (T244529) (duration: 01m 20s) |
[production] |
| 14:33 |
<vgutierrez> |
depool and reimage ncredir3001 as buster - T243391 |
[production] |
| 14:32 |
<vgutierrez> |
depool & reimage cp4031 as buster - T242093 |
[production] |
| 14:23 |
<vgutierrez> |
pooling ncredir3002 running buster - T243391 |
[production] |
| 13:26 |
<vgutierrez> |
pooling cp4021 with buster - T242093 |
[production] |
| 13:05 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:03 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:51 |
<vgutierrez> |
depool and reimage ncredir3002 as buster - T243391 |
[production] |
| 12:42 |
<vgutierrez> |
depool & reimage cp4021 as buster - T242093 |
[production] |
| 12:08 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:08 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:58 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:57 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:25 |
<vgutierrez> |
pooling ncredir5001 running buster - T243391 |
[production] |
| 11:24 |
<vgutierrez> |
pooling cp4022 with buster - T242093 |
[production] |
| 11:09 |
<akosiaris> |
undo wikifeeds experiments |
[production] |
| 11:07 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 10:42 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 10:40 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:37 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:36 |
<akosiaris> |
conduct experiments with stopping/starting uwsgi-ores on ores2001 T242705 |
[production] |
| 10:24 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 10:23 |
<vgutierrez> |
depool and reimage ncredir5001 as buster - T243391 |
[production] |
| 10:14 |
<vgutierrez> |
depool & reimage cp4022 as buster - T242093 |
[production] |
| 10:02 |
<akosiaris> |
increase capacity for wikifeeds by 50% T244535 |
[production] |
| 10:01 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 10:01 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 09:53 |
<ema> |
A:mw: increase keepalive_requests from 100 to 200 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/570670/ T241145 |
[production] |
| 09:09 |
<godog> |
roll restart cassandra instance on restbase-dev |
[production] |
| 09:03 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 09:03 |
<godog> |
restart cassandra on restbase-dev1004 to test logging pipeline onboard |
[production] |
| 09:01 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 08:59 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1090:3312, db1090:3317', diff saved to https://phabricator.wikimedia.org/P10343 and previous config saved to /var/cache/conftool/dbconfig/20200207-085846-marostegui.json |
[production] |
| 08:54 |
<marostegui> |
Upgrade db1090:3312, db1090:3317 |
[production] |
| 08:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1090:3312, db1090:3317 for upgrade', diff saved to https://phabricator.wikimedia.org/P10342 and previous config saved to /var/cache/conftool/dbconfig/20200207-085432-marostegui.json |
[production] |
| 08:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1101:3317 T239453', diff saved to https://phabricator.wikimedia.org/P10341 and previous config saved to /var/cache/conftool/dbconfig/20200207-084447-marostegui.json |
[production] |