5501-5550 of 10000 results (79ms)
2022-06-13 §
10:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1143 (T310011)', diff saved to https://phabricator.wikimedia.org/P29642 and previous config saved to /var/cache/conftool/dbconfig/20220613-104449-marostegui.json [production]
10:38 <klausman@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
10:38 <klausman@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
10:37 <klausman@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
10:37 <klausman@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
10:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1143 (T310011)', diff saved to https://phabricator.wikimedia.org/P29641 and previous config saved to /var/cache/conftool/dbconfig/20220613-101537-marostegui.json [production]
10:15 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1143.eqiad.wmnet with reason: Maintenance [production]
10:15 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1143.eqiad.wmnet with reason: Maintenance [production]
10:13 <moritzm> installing 5.10.120 kernel updates on bullseye hosts [production]
09:53 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
09:53 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
09:12 <moritzm> drain ganeti3001 for firmware update/reimage T308238 [production]
09:07 <moritzm> installing ntfs-3g security updates [production]
07:54 <moritzm> failover ganeti master in esams to ganeti3003 T308238 [production]
07:18 <joal> Manually rerun webrequest_text laod for hour 2022-06-12T08:00 [production]
06:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P29640 and previous config saved to /var/cache/conftool/dbconfig/20220613-064109-root.json [production]
06:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P29639 and previous config saved to /var/cache/conftool/dbconfig/20220613-062605-root.json [production]
06:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P29638 and previous config saved to /var/cache/conftool/dbconfig/20220613-061101-root.json [production]
05:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P29637 and previous config saved to /var/cache/conftool/dbconfig/20220613-055557-root.json [production]
05:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P29636 and previous config saved to /var/cache/conftool/dbconfig/20220613-054623-marostegui.json [production]
05:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P29635 and previous config saved to /var/cache/conftool/dbconfig/20220613-053118-marostegui.json [production]
05:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1172 (T310011)', diff saved to https://phabricator.wikimedia.org/P29634 and previous config saved to /var/cache/conftool/dbconfig/20220613-051613-marostegui.json [production]
05:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1172 (T310011)', diff saved to https://phabricator.wikimedia.org/P29633 and previous config saved to /var/cache/conftool/dbconfig/20220613-051407-marostegui.json [production]
05:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
05:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
05:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
05:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
04:59 <kart_> Updated cxserver to 2022-06-08-124326-production + nodejs > node command update (T306995, T309169) [production]
04:57 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
04:56 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
04:54 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
04:54 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
04:50 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
04:50 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
04:44 <dzahn@cumin2002> conftool action : set/pooled=inactive; selector: dc=codfw,name=thumbor2004.codfw.wmnet [production]
04:32 <dzahn@cumin2002> conftool action : set/pooled=no; selector: dc=codfw,name=thumbor2004.codfw.wmnet [production]
04:29 <mutante> thumbor2004 - attempted powercycle via DRAC console [production]
04:25 <mutante> thumbor2006 - host down - attempting powercycle via DRAC console [production]
02:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1105:3311 (T298560)', diff saved to https://phabricator.wikimedia.org/P29629 and previous config saved to /var/cache/conftool/dbconfig/20220613-021511-ladsgroup.json [production]
02:15 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
02:15 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
2022-06-12 §
18:31 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddumps1002.wikimedia.org with OS bullseye [production]
18:17 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddumps1002.wikimedia.org with reason: host reimage [production]
18:14 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on clouddumps1002.wikimedia.org with reason: host reimage [production]
18:03 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1002.wikimedia.org with OS bullseye [production]
14:52 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddumps1002.wikimedia.org with OS bullseye [production]
14:39 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddumps1002.wikimedia.org with reason: host reimage [production]
14:36 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on clouddumps1002.wikimedia.org with reason: host reimage [production]
14:25 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1002.wikimedia.org with OS bullseye [production]
13:16 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Revision table maint [production]