2751-2800 of 10000 results (89ms)
2023-05-08 §
08:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1167 (T335845)', diff saved to https://phabricator.wikimedia.org/P47785 and previous config saved to /var/cache/conftool/dbconfig/20230508-081353-ladsgroup.json [production]
08:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
08:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
08:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
08:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
08:13 <marostegui@deploy1002> Finished scap: Backport for [[gerrit:916900|ProductionServices.php: Promote pc1014 to pc1 master]] (duration: 15m 27s) [production]
07:59 <jmm@cumin2002> START - Cookbook sre.ganeti.reimage for host netflow2003.codfw.wmnet with OS bookworm [production]
07:59 <marostegui@deploy1002> marostegui: Backport for [[gerrit:916900|ProductionServices.php: Promote pc1014 to pc1 master]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
07:57 <marostegui@deploy1002> Started scap: Backport for [[gerrit:916900|ProductionServices.php: Promote pc1014 to pc1 master]] [production]
07:54 <_joe_> restarting varnish-frontend on cp5029, last host in eqsin/upload to be restarted [production]
07:53 <vgutierrez> fetch HAProxy 2.6.13 on thirdparty/haproxy2.6 (apt.wm.o) - T334448 [production]
07:51 <jmm@cumin2002> END (ERROR) - Cookbook sre.ganeti.reimage (exit_code=97) for host netflow2003.codfw.wmnet with OS bookworm [production]
07:29 <_joe_> restarting varnish-frontend on upload eqsin [production]
07:25 <_joe_> running restart-cdn on cp5030 [production]
07:22 <jmm@cumin2002> START - Cookbook sre.ganeti.reimage for host netflow2003.codfw.wmnet with OS bookworm [production]
07:22 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1001.eqiad.wmnet with OS bullseye [production]
07:19 <moritzm> updated bookworm installer to RC2 T330495 [production]
07:11 <jmm@cumin2002> END (ERROR) - Cookbook sre.ganeti.reimage (exit_code=97) for host netflow2003.codfw.wmnet with OS bookworm [production]
07:05 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage [production]
07:02 <volans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage [production]
06:50 <moritzm> bounce ferm on aux-k8s-ctrl1001 [production]
06:49 <jmm@cumin2002> START - Cookbook sre.ganeti.reimage for host netflow2003.codfw.wmnet with OS bookworm [production]
06:48 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.reimage (exit_code=99) for host netflow2003.codfw.wmnet with OS bookworm [production]
06:48 <volans@cumin1001> START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS bullseye [production]
06:48 <kart_> Deployed MinT to the production (T331505) [production]
06:47 <jmm@cumin2002> START - Cookbook sre.ganeti.reimage for host netflow2003.codfw.wmnet with OS bookworm [production]
06:47 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply [production]
06:44 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/machinetranslation: apply [production]
06:43 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply [production]
06:40 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/machinetranslation: apply [production]
05:55 <marostegui@deploy1002> Finished scap: Backport for [[gerrit:915725|Revert "ProductionServices.php: Promote pc2014 to pc1 master"]] (duration: 27m 46s) [production]
05:46 <phedenskog@deploy1002> Finished deploy [performance/navtiming@9b22d3b]: Measure largest contentful paint element type (duration: 00m 05s) [production]
05:46 <phedenskog@deploy1002> Started deploy [performance/navtiming@9b22d3b]: Measure largest contentful paint element type [production]
05:42 <marostegui@deploy1002> marostegui: Backport for [[gerrit:915725|Revert "ProductionServices.php: Promote pc2014 to pc1 master"]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
05:28 <marostegui@deploy1002> Started scap: Backport for [[gerrit:915725|Revert "ProductionServices.php: Promote pc2014 to pc1 master"]] [production]
05:18 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc1014.eqiad.wmnet with reason: Maintenance [production]
05:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on pc1014.eqiad.wmnet with reason: Maintenance [production]
05:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1113 (s5,s6) T336029', diff saved to https://phabricator.wikimedia.org/P47783 and previous config saved to /var/cache/conftool/dbconfig/20230508-051036-root.json [production]
05:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbproxy1013.eqiad.wmnet with reason: Maintenance [production]
05:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on dbproxy1013.eqiad.wmnet with reason: Maintenance [production]
04:54 <marostegui> Deploy schema change on x1 eqiad wikishared with replication dbmaint T335834 [production]
2023-05-07 §
00:54 <sukhe> restart haproxy on cp1087: T334448 [production]
2023-05-06 §
08:51 <jelto@cumin1001> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Install software version upgrade [production]
08:03 <jelto@cumin1001> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade [production]
07:50 <jelto@cumin1001> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Install software version upgrade [production]
07:44 <jelto@cumin1001> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Install software version upgrade [production]
07:07 <jelto@cumin1001> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade [production]
06:50 <jelto@cumin1001> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Install software version upgrade [production]
2023-05-05 §
23:24 <tzatziki> removing emails from 230 users per self-requests [production]
18:57 <brennen@deploy1002> Finished scap: Backport for [[gerrit:915719|Revert "api: Use RevisionStore::newRevisionsFromBatch to fetch revision records" (T336008 T336022)]] (duration: 14m 21s) [production]