451-500 of 10000 results (80ms)
2023-05-11 §
05:57 <marostegui@deploy1002> Started scap: Backport for [[gerrit:918903|ProductionServices.php: Failover pc2 codfw master]] [production]
05:56 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/machinetranslation: apply [production]
05:55 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply [production]
05:53 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/machinetranslation: apply [production]
05:48 <jynus@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2139.codfw.wmnet with reason: T335396 [production]
05:47 <jynus@cumin1001> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db2139.codfw.wmnet with reason: T335396 [production]
05:45 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/machinetranslation: apply [production]
05:44 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/machinetranslation: apply [production]
2023-05-10 §
22:08 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2021.codfw.wmnet with OS buster [production]
21:52 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [production]
21:49 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [production]
21:32 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS buster [production]
21:31 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2021.codfw.wmnet with OS buster [production]
21:31 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS buster [production]
20:58 <ejegg> payments-wiki upgraded from 2125cea7 to d1c5fefc [production]
20:58 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2021.codfw.wmnet with OS bullseye [production]
20:55 <milimetric@deploy1002> Finished deploy [airflow-dags/analytics@02d6ac9]: (no justification provided) (duration: 00m 11s) [production]
20:55 <milimetric@deploy1002> Started deploy [airflow-dags/analytics@02d6ac9]: (no justification provided) [production]
20:33 <hashar@deploy1002> Finished deploy [gerrit/gerrit@e815301]: Gerrit to 3.5.6 on gerrit1003 | T336339 (duration: 00m 06s) [production]
20:33 <hashar@deploy1002> Started deploy [gerrit/gerrit@e815301]: Gerrit to 3.5.6 on gerrit1003 | T336339 [production]
20:32 <cjming> end of UTC late backport window [production]
20:21 <cjming@deploy1002> Finished scap: Backport for [[gerrit:918531|Remove unnecessary jQuery closure (T324913)]] (duration: 09m 02s) [production]
20:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T335845)', diff saved to https://phabricator.wikimedia.org/P48177 and previous config saved to /var/cache/conftool/dbconfig/20230510-202014-ladsgroup.json [production]
20:14 <cjming@deploy1002> cjming and jdlrobson: Backport for [[gerrit:918531|Remove unnecessary jQuery closure (T324913)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
20:12 <cjming@deploy1002> Started scap: Backport for [[gerrit:918531|Remove unnecessary jQuery closure (T324913)]] [production]
20:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P48176 and previous config saved to /var/cache/conftool/dbconfig/20230510-200508-ladsgroup.json [production]
20:01 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS bullseye [production]
20:00 <milimetric@deploy1002> Finished deploy [analytics/refinery@4ccc172] (thin): Regular analytics weekly train THIN [analytics/refinery@4ccc172] (duration: 00m 05s) [production]
20:00 <milimetric@deploy1002> Started deploy [analytics/refinery@4ccc172] (thin): Regular analytics weekly train THIN [analytics/refinery@4ccc172] [production]
20:00 <milimetric@deploy1002> Finished deploy [analytics/refinery@4ccc172] (thin): Regular analytics weekly train THIN [analytics/refinery@4ccc172] (duration: 00m 26s) [production]
19:59 <milimetric@deploy1002> Started deploy [analytics/refinery@4ccc172] (thin): Regular analytics weekly train THIN [analytics/refinery@4ccc172] [production]
19:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P48175 and previous config saved to /var/cache/conftool/dbconfig/20230510-195001-ladsgroup.json [production]
19:47 <bking@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=wdqs,name=codfw [production]
19:35 <milimetric@deploy1002> Finished deploy [analytics/refinery@4ccc172]: Regular analytics weekly train [analytics/refinery@4ccc172] (duration: 40m 28s) [production]
19:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T335845)', diff saved to https://phabricator.wikimedia.org/P48174 and previous config saved to /var/cache/conftool/dbconfig/20230510-193455-ladsgroup.json [production]
19:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1219 (T335845)', diff saved to https://phabricator.wikimedia.org/P48173 and previous config saved to /var/cache/conftool/dbconfig/20230510-192746-ladsgroup.json [production]
19:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
19:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
19:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1218 (T335845)', diff saved to https://phabricator.wikimedia.org/P48172 and previous config saved to /var/cache/conftool/dbconfig/20230510-192722-ladsgroup.json [production]
19:25 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Rolling restart to apply Cassandra 3.11.14 upgrade - eevans@cumin1001 [production]
19:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P48171 and previous config saved to /var/cache/conftool/dbconfig/20230510-191216-ladsgroup.json [production]
19:08 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Rolling restart to apply Cassandra 3.11.14 upgrade - eevans@cumin1001 [production]
19:00 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Rolling restart to apply Cassandra 3.11.14 upgrade - eevans@cumin1001 [production]
18:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P48170 and previous config saved to /var/cache/conftool/dbconfig/20230510-185710-ladsgroup.json [production]
18:54 <milimetric@deploy1002> Started deploy [analytics/refinery@4ccc172]: Regular analytics weekly train [analytics/refinery@4ccc172] [production]
18:45 <sukhe@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: LVS reimaging in codfw, blocking deploys T326767 (duration: 191m 53s) [production]
18:43 <ejegg> payments-wiki upgraded from ec5a5e92 to 2125cea7 [production]
18:43 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Rolling restart to apply Cassandra 3.11.14 upgrade - eevans@cumin1001 [production]
18:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1218 (T335845)', diff saved to https://phabricator.wikimedia.org/P48169 and previous config saved to /var/cache/conftool/dbconfig/20230510-184202-ladsgroup.json [production]
18:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1218 (T335845)', diff saved to https://phabricator.wikimedia.org/P48168 and previous config saved to /var/cache/conftool/dbconfig/20230510-183441-ladsgroup.json [production]