3301-3350 of 10000 results (87ms)
2023-05-11 §
06:21 <marostegui@deploy1002> Started scap: Backport for [[gerrit:918534|Revert "ProductionServices.php: Failover pc2 codfw master"]] [production]
06:21 <XioNoX> Configure/reconfigure 1:1 NAT for new fr-tech hosts (frbast2002, frmon2002) - T336450 [production]
06:15 <ayounsi@cumin1001> END (ERROR) - Cookbook sre.network.peering (exit_code=97) with action 'configure' for AS: 13335 [production]
06:14 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 13335 [production]
06:11 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 714 [production]
06:07 <marostegui@deploy1002> Finished scap: Backport for [[gerrit:918903|ProductionServices.php: Failover pc2 codfw master]] (duration: 07m 42s) [production]
06:05 <kart_> Updated MinT to 2023-05-11-051736-production [production]
06:01 <marostegui@deploy1002> marostegui: Backport for [[gerrit:918903|ProductionServices.php: Failover pc2 codfw master]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
06:00 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply [production]
05:59 <marostegui@deploy1002> Started scap: Backport for [[gerrit:918903|ProductionServices.php: Failover pc2 codfw master]] [production]
05:58 <marostegui@deploy1002> marostegui: Backport for [[gerrit:918903|ProductionServices.php: Failover pc2 codfw master]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
05:58 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 714 [production]
05:57 <marostegui@deploy1002> Started scap: Backport for [[gerrit:918903|ProductionServices.php: Failover pc2 codfw master]] [production]
05:56 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/machinetranslation: apply [production]
05:55 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply [production]
05:53 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/machinetranslation: apply [production]
05:48 <jynus@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2139.codfw.wmnet with reason: T335396 [production]
05:47 <jynus@cumin1001> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db2139.codfw.wmnet with reason: T335396 [production]
05:45 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/machinetranslation: apply [production]
05:44 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/machinetranslation: apply [production]
2023-05-10 §
22:08 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2021.codfw.wmnet with OS buster [production]
21:52 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [production]
21:49 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [production]
21:32 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS buster [production]
21:31 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2021.codfw.wmnet with OS buster [production]
21:31 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS buster [production]
20:58 <ejegg> payments-wiki upgraded from 2125cea7 to d1c5fefc [production]
20:58 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2021.codfw.wmnet with OS bullseye [production]
20:55 <milimetric@deploy1002> Finished deploy [airflow-dags/analytics@02d6ac9]: (no justification provided) (duration: 00m 11s) [production]
20:55 <milimetric@deploy1002> Started deploy [airflow-dags/analytics@02d6ac9]: (no justification provided) [production]
20:33 <hashar@deploy1002> Finished deploy [gerrit/gerrit@e815301]: Gerrit to 3.5.6 on gerrit1003 | T336339 (duration: 00m 06s) [production]
20:33 <hashar@deploy1002> Started deploy [gerrit/gerrit@e815301]: Gerrit to 3.5.6 on gerrit1003 | T336339 [production]
20:32 <cjming> end of UTC late backport window [production]
20:21 <cjming@deploy1002> Finished scap: Backport for [[gerrit:918531|Remove unnecessary jQuery closure (T324913)]] (duration: 09m 02s) [production]
20:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T335845)', diff saved to https://phabricator.wikimedia.org/P48177 and previous config saved to /var/cache/conftool/dbconfig/20230510-202014-ladsgroup.json [production]
20:14 <cjming@deploy1002> cjming and jdlrobson: Backport for [[gerrit:918531|Remove unnecessary jQuery closure (T324913)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
20:12 <cjming@deploy1002> Started scap: Backport for [[gerrit:918531|Remove unnecessary jQuery closure (T324913)]] [production]
20:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P48176 and previous config saved to /var/cache/conftool/dbconfig/20230510-200508-ladsgroup.json [production]
20:01 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS bullseye [production]
20:00 <milimetric@deploy1002> Finished deploy [analytics/refinery@4ccc172] (thin): Regular analytics weekly train THIN [analytics/refinery@4ccc172] (duration: 00m 05s) [production]
20:00 <milimetric@deploy1002> Started deploy [analytics/refinery@4ccc172] (thin): Regular analytics weekly train THIN [analytics/refinery@4ccc172] [production]
20:00 <milimetric@deploy1002> Finished deploy [analytics/refinery@4ccc172] (thin): Regular analytics weekly train THIN [analytics/refinery@4ccc172] (duration: 00m 26s) [production]
19:59 <milimetric@deploy1002> Started deploy [analytics/refinery@4ccc172] (thin): Regular analytics weekly train THIN [analytics/refinery@4ccc172] [production]
19:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P48175 and previous config saved to /var/cache/conftool/dbconfig/20230510-195001-ladsgroup.json [production]
19:47 <bking@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=wdqs,name=codfw [production]
19:35 <milimetric@deploy1002> Finished deploy [analytics/refinery@4ccc172]: Regular analytics weekly train [analytics/refinery@4ccc172] (duration: 40m 28s) [production]
19:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T335845)', diff saved to https://phabricator.wikimedia.org/P48174 and previous config saved to /var/cache/conftool/dbconfig/20230510-193455-ladsgroup.json [production]
19:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1219 (T335845)', diff saved to https://phabricator.wikimedia.org/P48173 and previous config saved to /var/cache/conftool/dbconfig/20230510-192746-ladsgroup.json [production]
19:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
19:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]