2301-2350 of 10000 results (83ms)
2022-09-29 §
05:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P35112 and previous config saved to /var/cache/conftool/dbconfig/20220929-051237-ladsgroup.json [production]
05:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s4 T318886 [production]
05:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db2140 with weight 0 T318886', diff saved to https://phabricator.wikimedia.org/P35111 and previous config saved to /var/cache/conftool/dbconfig/20220929-051114-root.json [production]
05:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s4 T318886 [production]
04:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P35110 and previous config saved to /var/cache/conftool/dbconfig/20220929-045730-ladsgroup.json [production]
04:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134 (T314041)', diff saved to https://phabricator.wikimedia.org/P35109 and previous config saved to /var/cache/conftool/dbconfig/20220929-044224-ladsgroup.json [production]
03:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2173 (T314041)', diff saved to https://phabricator.wikimedia.org/P35108 and previous config saved to /var/cache/conftool/dbconfig/20220929-035724-ladsgroup.json [production]
03:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
03:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
03:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance [production]
03:56 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance [production]
03:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T314041)', diff saved to https://phabricator.wikimedia.org/P35107 and previous config saved to /var/cache/conftool/dbconfig/20220929-035647-ladsgroup.json [production]
03:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P35106 and previous config saved to /var/cache/conftool/dbconfig/20220929-034140-ladsgroup.json [production]
03:40 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 10s) [production]
03:40 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b9be20d]: (no justification provided) [production]
03:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3311', diff saved to https://phabricator.wikimedia.org/P35105 and previous config saved to /var/cache/conftool/dbconfig/20220929-032634-ladsgroup.json [production]
03:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3311 (T314041)', diff saved to https://phabricator.wikimedia.org/P35104 and previous config saved to /var/cache/conftool/dbconfig/20220929-031127-ladsgroup.json [production]
02:29 <ejegg> updated fundraising CiviCRM from f3461a44 to 5e1738a1 [production]
02:20 <ejegg> updated fundraising python tools from dd494413 to 14d60435 [production]
01:01 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash2037.codfw.wmnet with OS buster [production]
00:46 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2037.codfw.wmnet with reason: host reimage [production]
00:43 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2037.codfw.wmnet with reason: host reimage [production]
2022-09-28 §
23:53 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host logstash2037.codfw.wmnet with OS buster [production]
23:52 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['logstash2037'] [production]
23:51 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['logstash2037'] [production]
23:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1134 (T314041)', diff saved to https://phabricator.wikimedia.org/P35103 and previous config saved to /var/cache/conftool/dbconfig/20220928-231719-ladsgroup.json [production]
23:17 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
23:17 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
22:20 <ejegg> updated fundraising CiviCRM from d31c19a0 to f3461a44 [production]
21:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2170:3311 (T314041)', diff saved to https://phabricator.wikimedia.org/P35102 and previous config saved to /var/cache/conftool/dbconfig/20220928-213701-ladsgroup.json [production]
21:36 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
21:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
21:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T314041)', diff saved to https://phabricator.wikimedia.org/P35101 and previous config saved to /var/cache/conftool/dbconfig/20220928-213640-ladsgroup.json [production]
21:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P35100 and previous config saved to /var/cache/conftool/dbconfig/20220928-212131-ladsgroup.json [production]
21:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P35099 and previous config saved to /var/cache/conftool/dbconfig/20220928-210624-ladsgroup.json [production]
21:06 <volans> installed spicerack 4.0.0-1+deb11u1 on cumin1001 [production]
20:59 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:57 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
20:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T314041)', diff saved to https://phabricator.wikimedia.org/P35098 and previous config saved to /var/cache/conftool/dbconfig/20220928-205117-ladsgroup.json [production]
20:50 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 12200 [production]
20:50 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 12200 [production]
20:39 <TheresNoTime> closing UTC late backport window [production]
20:27 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:26 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:26 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:25 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:24 <samtar@deploy1002> Finished scap: Backport for [[gerrit:836244|[config]: Deploy GDI survey Wave 3 (T318156)]] (duration: 06m 19s) [production]
20:20 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:19 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]