5501-5550 of 10000 results (91ms)
2022-12-07 ยง
17:01 <jiji@deploy1002> Finished scap: Backport for [[gerrit:865123|ProductionServices: Use redis_misc servers for LockManager (6/6) (T267581)]] (duration: 14m 46s) [production]
16:58 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1023.eqiad.wmnet with OS bullseye [production]
16:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P42458 and previous config saved to /var/cache/conftool/dbconfig/20221207-165815-ladsgroup.json [production]
16:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P42457 and previous config saved to /var/cache/conftool/dbconfig/20221207-165758-ladsgroup.json [production]
16:57 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1024.eqiad.wmnet with OS bullseye [production]
16:56 <eevans@deploy1002> helmfile [codfw] DONE helmfile.d/services/echostore: apply [production]
16:55 <eevans@deploy1002> helmfile [codfw] START helmfile.d/services/echostore: apply [production]
16:55 <eevans@deploy1002> helmfile [staging] DONE helmfile.d/services/echostore: apply [production]
16:55 <eevans@deploy1002> helmfile [staging] START helmfile.d/services/echostore: apply [production]
16:48 <jiji@deploy1002> jiji and jiji: Backport for [[gerrit:865123|ProductionServices: Use redis_misc servers for LockManager (6/6) (T267581)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
16:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181 (T322618)', diff saved to https://phabricator.wikimedia.org/P42456 and previous config saved to /var/cache/conftool/dbconfig/20221207-164809-ladsgroup.json [production]
16:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P42455 and previous config saved to /var/cache/conftool/dbconfig/20221207-164748-ladsgroup.json [production]
16:46 <jiji@deploy1002> Started scap: Backport for [[gerrit:865123|ProductionServices: Use redis_misc servers for LockManager (6/6) (T267581)]] [production]
16:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P42454 and previous config saved to /var/cache/conftool/dbconfig/20221207-164308-ladsgroup.json [production]
16:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1181 (T322618)', diff saved to https://phabricator.wikimedia.org/P42453 and previous config saved to /var/cache/conftool/dbconfig/20221207-164258-ladsgroup.json [production]
16:42 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
16:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P42452 and previous config saved to /var/cache/conftool/dbconfig/20221207-164252-ladsgroup.json [production]
16:42 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs5002.eqsin.wmnet with reason: downtimed, in the process of decom [production]
16:42 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
16:42 <sukhe> restart pybal on lvs5002 [production]
16:42 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on lvs5002.eqsin.wmnet with reason: downtimed, in the process of decom [production]
16:38 <sukhe> cr[23]-eqsin*: set routing-options static route 103.102.166.240/28 next-hop 10.132.0.6: T322048 [production]
16:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2173 (T322618)', diff saved to https://phabricator.wikimedia.org/P42451 and previous config saved to /var/cache/conftool/dbconfig/20221207-163242-ladsgroup.json [production]
16:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2173 (T322618)', diff saved to https://phabricator.wikimedia.org/P42450 and previous config saved to /var/cache/conftool/dbconfig/20221207-163031-ladsgroup.json [production]
16:30 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
16:30 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
16:30 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2173.codfw.wmnet with reason: Maintenance [production]
16:29 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2173.codfw.wmnet with reason: Maintenance [production]
16:29 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1026.eqiad.wmnet with OS bullseye [production]
16:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P42449 and previous config saved to /var/cache/conftool/dbconfig/20221207-162802-ladsgroup.json [production]
16:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P42448 and previous config saved to /var/cache/conftool/dbconfig/20221207-162745-ladsgroup.json [production]
16:27 <aqu@deploy1002> Started deploy [analytics/refinery@349e1cc]: Deploy HDFS usage dataset generation scripts [analytics/refinery@349e1cc] [production]
16:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1096:3316 (T322618)', diff saved to https://phabricator.wikimedia.org/P42447 and previous config saved to /var/cache/conftool/dbconfig/20221207-162553-ladsgroup.json [production]
16:25 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
16:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2117 (T322618)', diff saved to https://phabricator.wikimedia.org/P42446 and previous config saved to /var/cache/conftool/dbconfig/20221207-162533-ladsgroup.json [production]
16:25 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
16:25 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
16:25 <aqu> Deploying analytics/refinery (HDFS usage scripts) [production]
16:25 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
16:24 <sukhe> run homer in cr*-eqsin for Gerrit: 865615 [production]
16:09 <denisse> sync rancid in netmon2001 and netmon2002 [production]
16:08 <denisse> sync librenms RRD in netmon2002 [production]
16:08 <jiji@deploy1002> Finished scap: Backport for [[gerrit:865122|ProductionServices: Use redis_misc servers for LockManager (5/6) (T267581)]] (duration: 10m 59s) [production]
16:06 <sukhe> run homer in cr*-eqsin for Gerrit: 865660 [production]
16:02 <denisse> Sync LibreNMS RRD in netmon2001 [production]
15:59 <jiji@deploy1002> jiji and jiji: Backport for [[gerrit:865122|ProductionServices: Use redis_misc servers for LockManager (5/6) (T267581)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
15:57 <jiji@deploy1002> Started scap: Backport for [[gerrit:865122|ProductionServices: Use redis_misc servers for LockManager (5/6) (T267581)]] [production]
15:46 <jiji@deploy1002> Finished scap: Backport for [[gerrit:865121|ProductionServices: Use redis_misc servers for LockManager (4/6) (T267581)]] (duration: 08m 29s) [production]
15:41 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5005.eqsin.wmnet with OS buster [production]
15:41 <sukhe@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin2002" [production]