|
2022-12-07
ยง
|
| 16:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P42457 and previous config saved to /var/cache/conftool/dbconfig/20221207-165758-ladsgroup.json |
[production] |
| 16:57 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubernetes1024.eqiad.wmnet with OS bullseye |
[production] |
| 16:56 |
<eevans@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/echostore: apply |
[production] |
| 16:55 |
<eevans@deploy1002> |
helmfile [codfw] START helmfile.d/services/echostore: apply |
[production] |
| 16:55 |
<eevans@deploy1002> |
helmfile [staging] DONE helmfile.d/services/echostore: apply |
[production] |
| 16:55 |
<eevans@deploy1002> |
helmfile [staging] START helmfile.d/services/echostore: apply |
[production] |
| 16:48 |
<jiji@deploy1002> |
jiji and jiji: Backport for [[gerrit:865123|ProductionServices: Use redis_misc servers for LockManager (6/6) (T267581)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 16:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T322618)', diff saved to https://phabricator.wikimedia.org/P42456 and previous config saved to /var/cache/conftool/dbconfig/20221207-164809-ladsgroup.json |
[production] |
| 16:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P42455 and previous config saved to /var/cache/conftool/dbconfig/20221207-164748-ladsgroup.json |
[production] |
| 16:46 |
<jiji@deploy1002> |
Started scap: Backport for [[gerrit:865123|ProductionServices: Use redis_misc servers for LockManager (6/6) (T267581)]] |
[production] |
| 16:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P42454 and previous config saved to /var/cache/conftool/dbconfig/20221207-164308-ladsgroup.json |
[production] |
| 16:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1181 (T322618)', diff saved to https://phabricator.wikimedia.org/P42453 and previous config saved to /var/cache/conftool/dbconfig/20221207-164258-ladsgroup.json |
[production] |
| 16:42 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P42452 and previous config saved to /var/cache/conftool/dbconfig/20221207-164252-ladsgroup.json |
[production] |
| 16:42 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs5002.eqsin.wmnet with reason: downtimed, in the process of decom |
[production] |
| 16:42 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:42 |
<sukhe> |
restart pybal on lvs5002 |
[production] |
| 16:42 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on lvs5002.eqsin.wmnet with reason: downtimed, in the process of decom |
[production] |
| 16:38 |
<sukhe> |
cr[23]-eqsin*: set routing-options static route 103.102.166.240/28 next-hop 10.132.0.6: T322048 |
[production] |
| 16:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173 (T322618)', diff saved to https://phabricator.wikimedia.org/P42451 and previous config saved to /var/cache/conftool/dbconfig/20221207-163242-ladsgroup.json |
[production] |
| 16:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2173 (T322618)', diff saved to https://phabricator.wikimedia.org/P42450 and previous config saved to /var/cache/conftool/dbconfig/20221207-163031-ladsgroup.json |
[production] |
| 16:30 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 16:30 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 16:30 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2173.codfw.wmnet with reason: Maintenance |
[production] |
| 16:29 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2173.codfw.wmnet with reason: Maintenance |
[production] |
| 16:29 |
<cwhite@cumin2002> |
START - Cookbook sre.hosts.reimage for host logstash1026.eqiad.wmnet with OS bullseye |
[production] |
| 16:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P42449 and previous config saved to /var/cache/conftool/dbconfig/20221207-162802-ladsgroup.json |
[production] |
| 16:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P42448 and previous config saved to /var/cache/conftool/dbconfig/20221207-162745-ladsgroup.json |
[production] |
| 16:27 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@349e1cc]: Deploy HDFS usage dataset generation scripts [analytics/refinery@349e1cc] |
[production] |
| 16:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P42447 and previous config saved to /var/cache/conftool/dbconfig/20221207-162553-ladsgroup.json |
[production] |
| 16:25 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P42446 and previous config saved to /var/cache/conftool/dbconfig/20221207-162533-ladsgroup.json |
[production] |
| 16:25 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance |
[production] |
| 16:25 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:25 |
<aqu> |
Deploying analytics/refinery (HDFS usage scripts) |
[production] |
| 16:25 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance |
[production] |
| 16:24 |
<sukhe> |
run homer in cr*-eqsin for Gerrit: 865615 |
[production] |
| 16:09 |
<denisse> |
sync rancid in netmon2001 and netmon2002 |
[production] |
| 16:08 |
<denisse> |
sync librenms RRD in netmon2002 |
[production] |
| 16:08 |
<jiji@deploy1002> |
Finished scap: Backport for [[gerrit:865122|ProductionServices: Use redis_misc servers for LockManager (5/6) (T267581)]] (duration: 10m 59s) |
[production] |
| 16:06 |
<sukhe> |
run homer in cr*-eqsin for Gerrit: 865660 |
[production] |
| 16:02 |
<denisse> |
Sync LibreNMS RRD in netmon2001 |
[production] |
| 15:59 |
<jiji@deploy1002> |
jiji and jiji: Backport for [[gerrit:865122|ProductionServices: Use redis_misc servers for LockManager (5/6) (T267581)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 15:57 |
<jiji@deploy1002> |
Started scap: Backport for [[gerrit:865122|ProductionServices: Use redis_misc servers for LockManager (5/6) (T267581)]] |
[production] |
| 15:46 |
<jiji@deploy1002> |
Finished scap: Backport for [[gerrit:865121|ProductionServices: Use redis_misc servers for LockManager (4/6) (T267581)]] (duration: 08m 29s) |
[production] |
| 15:41 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5005.eqsin.wmnet with OS buster |
[production] |
| 15:41 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin2002" |
[production] |
| 15:40 |
<sukhe@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin2002" |
[production] |
| 15:39 |
<jiji@deploy1002> |
jiji and jiji: Backport for [[gerrit:865121|ProductionServices: Use redis_misc servers for LockManager (4/6) (T267581)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 15:37 |
<jiji@deploy1002> |
Started scap: Backport for [[gerrit:865121|ProductionServices: Use redis_misc servers for LockManager (4/6) (T267581)]] |
[production] |