|
2022-11-10
ยง
|
| 19:05 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P39070 and previous config saved to /var/cache/conftool/dbconfig/20221110-190424-ladsgroup.json |
[production] |
| 19:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39069 and previous config saved to /var/cache/conftool/dbconfig/20221110-190353-ladsgroup.json |
[production] |
| 19:03 |
<dzahn@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:58 |
<mutante> |
phabricator - running decom cookbook on phab2001 - T322250 |
[production] |
| 18:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39068 and previous config saved to /var/cache/conftool/dbconfig/20221110-185640-ladsgroup.json |
[production] |
| 18:55 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts phab2001.codfw.wmnet |
[production] |
| 18:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39067 and previous config saved to /var/cache/conftool/dbconfig/20221110-185450-marostegui.json |
[production] |
| 18:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P39066 and previous config saved to /var/cache/conftool/dbconfig/20221110-185202-marostegui.json |
[production] |
| 18:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2171:3315 (T321130)', diff saved to https://phabricator.wikimedia.org/P39065 and previous config saved to /var/cache/conftool/dbconfig/20221110-185135-marostegui.json |
[production] |
| 18:51 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance |
[production] |
| 18:51 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db2171.codfw.wmnet with reason: Maintenance |
[production] |
| 18:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39064 and previous config saved to /var/cache/conftool/dbconfig/20221110-185103-marostegui.json |
[production] |
| 18:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P39063 and previous config saved to /var/cache/conftool/dbconfig/20221110-184917-ladsgroup.json |
[production] |
| 18:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P39062 and previous config saved to /var/cache/conftool/dbconfig/20221110-184847-ladsgroup.json |
[production] |
| 18:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P39061 and previous config saved to /var/cache/conftool/dbconfig/20221110-184133-ladsgroup.json |
[production] |
| 18:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39060 and previous config saved to /var/cache/conftool/dbconfig/20221110-183655-marostegui.json |
[production] |
| 18:36 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs4008.ulsfo.wmnet with reason: downtimed as we are resolving issues with LVS configuration |
[production] |
| 18:36 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs4008.ulsfo.wmnet with reason: downtimed as we are resolving issues with LVS configuration |
[production] |
| 18:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P39059 and previous config saved to /var/cache/conftool/dbconfig/20221110-183556-marostegui.json |
[production] |
| 18:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1169 (T321123)', diff saved to https://phabricator.wikimedia.org/P39058 and previous config saved to /var/cache/conftool/dbconfig/20221110-183548-marostegui.json |
[production] |
| 18:35 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:35 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:35 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:35 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:35 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:34 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1139.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135 (T321123)', diff saved to https://phabricator.wikimedia.org/P39057 and previous config saved to /var/cache/conftool/dbconfig/20221110-183455-marostegui.json |
[production] |
| 18:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P39056 and previous config saved to /var/cache/conftool/dbconfig/20221110-183340-ladsgroup.json |
[production] |
| 18:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P39055 and previous config saved to /var/cache/conftool/dbconfig/20221110-182627-ladsgroup.json |
[production] |
| 18:26 |
<ladsgroup@deploy1002> |
Synchronized portals: (no justification provided) (duration: 03m 38s) |
[production] |
| 18:22 |
<ladsgroup@deploy1002> |
Synchronized portals/wikipedia.org/assets: (no justification provided) (duration: 03m 46s) |
[production] |
| 18:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P39054 and previous config saved to /var/cache/conftool/dbconfig/20221110-182049-marostegui.json |
[production] |
| 18:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P39053 and previous config saved to /var/cache/conftool/dbconfig/20221110-181948-marostegui.json |
[production] |
| 18:18 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:18 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:855666|Bump portals to HEAD (T273179)]] (duration: 05m 14s) |
[production] |
| 18:17 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:17 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:17 |
<dcausse@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@a030f5f]: T320656: convert_to_esbulk: fix typo in config (duration: 02m 22s) |
[production] |
| 18:16 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:15 |
<dcausse@deploy1002> |
Started deploy [wikimedia/discovery/analytics@a030f5f]: T320656: convert_to_esbulk: fix typo in config |
[production] |
| 18:13 |
<ladsgroup@deploy1002> |
ladsgroup and ladsgroup: Backport for [[gerrit:855666|Bump portals to HEAD (T273179)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 18:12 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:855666|Bump portals to HEAD (T273179)]] |
[production] |
| 18:09 |
<volans> |
upgrading spicerack to 5.0.0 on cumin hosts |
[production] |
| 18:05 |
<volans> |
uploaded spicerack_5.0.0 to apt.wikimedia.org bullseye-wikimedia |
[production] |
| 18:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39052 and previous config saved to /var/cache/conftool/dbconfig/20221110-180543-marostegui.json |
[production] |
| 18:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P39051 and previous config saved to /var/cache/conftool/dbconfig/20221110-180442-marostegui.json |
[production] |
| 18:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2157 (T321130)', diff saved to https://phabricator.wikimedia.org/P39050 and previous config saved to /var/cache/conftool/dbconfig/20221110-180228-marostegui.json |
[production] |
| 18:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 18:02 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |