1701-1750 of 10000 results (74ms)
2022-11-29 ยง
10:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119 (T321126)', diff saved to https://phabricator.wikimedia.org/P41677 and previous config saved to /var/cache/conftool/dbconfig/20221129-105050-marostegui.json [production]
10:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P41676 and previous config saved to /var/cache/conftool/dbconfig/20221129-105030-ladsgroup.json [production]
10:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P41675 and previous config saved to /var/cache/conftool/dbconfig/20221129-105011-ladsgroup.json [production]
10:49 <oblivian@puppetmaster1001> conftool action : set/weight=30; selector: cluster=api_appserver,dc=eqiad,name=mw14[6-9].* [production]
10:48 <oblivian@puppetmaster1001> conftool action : set/weight=30; selector: cluster=appserver,dc=eqiad,name=mw14[7-9].* [production]
10:48 <hnowlan> stopping puppet on maps* for casssandra removal [production]
10:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1119 (T321126)', diff saved to https://phabricator.wikimedia.org/P41674 and previous config saved to /var/cache/conftool/dbconfig/20221129-104828-marostegui.json [production]
10:48 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
10:48 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
10:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118 (T321126)', diff saved to https://phabricator.wikimedia.org/P41673 and previous config saved to /var/cache/conftool/dbconfig/20221129-104807-marostegui.json [production]
10:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P41672 and previous config saved to /var/cache/conftool/dbconfig/20221129-103852-ladsgroup.json [production]
10:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P41671 and previous config saved to /var/cache/conftool/dbconfig/20221129-103524-ladsgroup.json [production]
10:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P41670 and previous config saved to /var/cache/conftool/dbconfig/20221129-103505-ladsgroup.json [production]
10:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P41669 and previous config saved to /var/cache/conftool/dbconfig/20221129-103301-marostegui.json [production]
10:30 <jynus> revoke temporary grants to scholarships for backups on db1117, db2160 T243037 [production]
10:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2156 (T322618)', diff saved to https://phabricator.wikimedia.org/P41668 and previous config saved to /var/cache/conftool/dbconfig/20221129-102746-ladsgroup.json [production]
10:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
10:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
10:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
10:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
10:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P41667 and previous config saved to /var/cache/conftool/dbconfig/20221129-102731-ladsgroup.json [production]
10:26 <elukey> restart kube-apiserver on ml-serve-ctrl* to clear out some knative controller issue [production]
10:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2120 (T323907)', diff saved to https://phabricator.wikimedia.org/P41666 and previous config saved to /var/cache/conftool/dbconfig/20221129-102345-ladsgroup.json [production]
10:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194 (T323907)', diff saved to https://phabricator.wikimedia.org/P41665 and previous config saved to /var/cache/conftool/dbconfig/20221129-101958-ladsgroup.json [production]
10:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P41664 and previous config saved to /var/cache/conftool/dbconfig/20221129-101754-marostegui.json [production]
10:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db2145 (re)pooling @ 100%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P41663 and previous config saved to /var/cache/conftool/dbconfig/20221129-101554-root.json [production]
10:15 <moritzm> upgrading puppetdb2003 to bookworm T321783 [production]
10:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P41662 and previous config saved to /var/cache/conftool/dbconfig/20221129-101225-ladsgroup.json [production]
10:07 <jynus> add temporary grants to scholarships for backups on db1117, db2160 T243037 [production]
10:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1194 (T323907)', diff saved to https://phabricator.wikimedia.org/P41661 and previous config saved to /var/cache/conftool/dbconfig/20221129-100319-ladsgroup.json [production]
10:03 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance [production]
10:03 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance [production]
10:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191 (T323907)', diff saved to https://phabricator.wikimedia.org/P41660 and previous config saved to /var/cache/conftool/dbconfig/20221129-100258-ladsgroup.json [production]
10:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118 (T321126)', diff saved to https://phabricator.wikimedia.org/P41659 and previous config saved to /var/cache/conftool/dbconfig/20221129-100248-marostegui.json [production]
10:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db2145 (re)pooling @ 75%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P41658 and previous config saved to /var/cache/conftool/dbconfig/20221129-100049-root.json [production]
10:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1118 (T321126)', diff saved to https://phabricator.wikimedia.org/P41657 and previous config saved to /var/cache/conftool/dbconfig/20221129-100025-marostegui.json [production]
10:00 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
09:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
09:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321126)', diff saved to https://phabricator.wikimedia.org/P41656 and previous config saved to /var/cache/conftool/dbconfig/20221129-095931-marostegui.json [production]
09:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P41655 and previous config saved to /var/cache/conftool/dbconfig/20221129-095718-ladsgroup.json [production]
09:56 <moritzm> installing curl security updates [production]
09:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2120 (T323907)', diff saved to https://phabricator.wikimedia.org/P41654 and previous config saved to /var/cache/conftool/dbconfig/20221129-094818-ladsgroup.json [production]
09:48 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance [production]
09:48 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance [production]
09:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2108 (T323907)', diff saved to https://phabricator.wikimedia.org/P41653 and previous config saved to /var/cache/conftool/dbconfig/20221129-094757-ladsgroup.json [production]
09:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P41652 and previous config saved to /var/cache/conftool/dbconfig/20221129-094745-ladsgroup.json [production]
09:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db2145 (re)pooling @ 50%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P41651 and previous config saved to /var/cache/conftool/dbconfig/20221129-094544-root.json [production]
09:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P41650 and previous config saved to /var/cache/conftool/dbconfig/20221129-094424-marostegui.json [production]
09:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P41649 and previous config saved to /var/cache/conftool/dbconfig/20221129-094212-ladsgroup.json [production]
09:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P41648 and previous config saved to /var/cache/conftool/dbconfig/20221129-093420-ladsgroup.json [production]