1551-1600 of 10000 results (57ms)
2022-07-11 §
08:06 <godog> trim thanos raw samples retention to 54w - T311690 [production]
08:04 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti2027.codfw.wmnet to cluster codfw and group A [production]
07:58 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet [production]
07:52 <godog> roll-restart swift-account swift-container across swift/thanos bullseye hosts - T297959 [production]
07:51 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:49 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet [production]
07:47 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:47 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:43 <taavi@deploy1002> Synchronized php-1.39.0-wmf.19/extensions/PageTriage/includes/HookHandlers/UndeleteHookHandler.php: Backport: [[gerrit:812532|UndeleteHookHandler: fix namespace conditional (T311347)]] (duration: 02m 54s) [production]
07:41 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2027.codfw.wmnet with OS bullseye [production]
07:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db2080 from dbtcl T312618', diff saved to https://phabricator.wikimedia.org/P30999 and previous config saved to /var/cache/conftool/dbconfig/20220711-073346-marostegui.json [production]
07:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2080.codfw.wmnet [production]
07:30 <marostegui@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:26 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2027.codfw.wmnet with reason: host reimage [production]
07:26 <marostegui@cumin1001> START - Cookbook sre.dns.netbox [production]
07:23 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2027.codfw.wmnet with reason: host reimage [production]
07:22 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db2080.codfw.wmnet [production]
07:09 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2027.codfw.wmnet with OS bullseye [production]
07:00 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2077.codfw.wmnet [production]
06:58 <marostegui@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:54 <marostegui@cumin1001> START - Cookbook sre.dns.netbox [production]
06:50 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db2077.codfw.wmnet [production]
06:28 <_joe_> repool thumbor1005 [production]
06:28 <_joe_> depooled thumbor1005, downgraded firejail, restarted units [production]
00:23 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
00:19 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
00:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
2022-07-10 §
13:48 <godog> silence ProbeDown pages for thumbor:8800 until wed [production]
2022-07-09 §
13:34 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:33 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:48 <krinkle@deploy1002> Synchronized php-1.39.0-wmf.19/includes/ResourceLoader/: I3e43b10d26858c5b (duration: 03m 37s) [production]
01:44 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
01:43 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
01:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
01:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:37 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
01:36 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
01:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
01:35 <krinkle@deploy1002> Synchronized wmf-config/: I1bb97d1d601 (duration: 03m 24s) [production]
01:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
2022-07-08 §
21:44 <ryankemper> [Elastic] Reshuffled shards on eqiad to get cluster back into green status (from yellow): https://phabricator.wikimedia.org/P30995#130117 [production]
21:32 <ori> apt1001: reprepro -C main include buster-wikimedia libvmod-querysort_0.2_amd64.changes [production]
19:58 <thcipriani> quick phab downtime for deploy to fix T312614 [production]
19:57 <cdanis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab.wmfusercontent.org with reason: bug fix [production]
19:57 <cdanis@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on phab.wmfusercontent.org with reason: bug fix [production]
19:57 <cdanis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phabricator.wikimedia.org with reason: bug fix [production]
19:56 <cdanis@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on phabricator.wikimedia.org with reason: bug fix [production]