2151-2200 of 10000 results (36ms)
2021-02-03 §
06:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1174 with minimal weight for the first time in s7', diff saved to https://phabricator.wikimedia.org/P14138 and previous config saved to /var/cache/conftool/dbconfig/20210203-063812-marostegui.json [production]
00:16 <jhuneidi@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
00:13 <legoktm@deploy1001> Synchronized logos/: Update and recompress logos for nlwiki, eswiki, ptwiki, ruwiki, svwiki, zhwiki (2/2) (duration: 01m 05s) [production]
00:12 <legoktm@deploy1001> Synchronized static/images/project-logos/: Update and recompress logos for nlwiki, eswiki, ptwiki, ruwiki, svwiki, zhwiki (1/2) (duration: 01m 10s) [production]
00:10 <jhuneidi@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
2021-02-02 §
23:53 <mutante> mw1300 - scap pull (it crashed earlier put is back after powercycling) [production]
23:52 <jhuneidi@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
23:30 <mutante> powercycling crashed m1300.eqiad.wmnet [production]
21:56 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1335.eqiad.wmnet [production]
21:56 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1336.eqiad.wmnet [production]
21:56 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1335.eqiad.wmnet [production]
21:55 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1336.eqiad.wmnet [production]
21:09 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1335.eqiad.wmnet with reason: REIMAGE [production]
21:07 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1336.eqiad.wmnet with reason: REIMAGE [production]
21:06 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1335.eqiad.wmnet with reason: REIMAGE [production]
21:05 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1336.eqiad.wmnet with reason: REIMAGE [production]
20:12 <cdanis> ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕒☕ sudo cumin A:cp 'enable-puppet "cdanis deploying I7003b7b6 and Idd0e124f5 T263496"' # test on cp2027 looks good, perhaps slightly-increased Varnish CPU consumption but hard to be sure [production]
20:00 <Lucas_WMDE> Morning backport window done [production]
19:58 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.36.0-wmf.29/extensions/WikibaseMediaInfo/: Backport: [[gerrit:661092|Pass $databaseName into WikiPageEntityDataLoader (T273622)]] (duration: 01m 07s) [production]
19:57 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.36.0-wmf.29/extensions/Wikibase/: Backport: [[gerrit:661091|Add wiki ID to WikiPageEntityDataLoader (T273622)]] (duration: 01m 25s) [production]
19:52 <cdanis> ❌cdanis@cumin1001.eqiad.wmnet ~ 🕒☕ sudo cumin A:cp 'disable-puppet "cdanis deploying I7003b7b6 and Idd0e124f5 T263496"' [production]
19:00 <mbsantos@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
18:48 <mbsantos@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
18:43 <mbsantos@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
18:23 <milimetric@deploy1001> Finished deploy [analytics/turnilo/deploy@052348b]: (no justification provided) (duration: 00m 03s) [production]
18:23 <milimetric@deploy1001> Started deploy [analytics/turnilo/deploy@052348b]: (no justification provided) [production]
18:22 <milimetric@deploy1001> deploy aborted: (no justification provided) (duration: 00m 10s) [production]
18:22 <milimetric@deploy1001> Started deploy [analytics/turnilo/deploy@052348b]: (no justification provided) [production]
18:17 <mbsantos@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
18:07 <mbsantos@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
18:03 <mbsantos@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
16:37 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host auth2001.codfw.wmnet [production]
16:33 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host auth1002.eqiad.wmnet [production]
16:30 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host auth1002.eqiad.wmnet [production]
16:30 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host auth2001.codfw.wmnet [production]
15:20 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host miscweb2002.codfw.wmnet [production]
15:19 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moscovium.eqiad.wmnet [production]
15:16 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host miscweb2002.codfw.wmnet [production]
15:16 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host moscovium.eqiad.wmnet [production]
14:39 <marostegui@cumin1001> dbctl commit (dc=all): 'db1094 (re)pooling @ 100%: Repool db1094 after cloning another host', diff saved to https://phabricator.wikimedia.org/P14135 and previous config saved to /var/cache/conftool/dbconfig/20210202-143950-root.json [production]
14:38 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1001.eqiad.wmnet [production]
14:35 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host failoid1001.eqiad.wmnet [production]
14:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1002.eqiad.wmnet [production]
14:35 <klausman@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ml-etcd2003.codfw.wmnet [production]
14:29 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2002.codfw.wmnet [production]
14:26 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host debmonitor1002.eqiad.wmnet [production]
14:26 <hashar@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.29 (duration: 73m 10s) [production]
14:26 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host debmonitor2002.codfw.wmnet [production]
14:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db1094 (re)pooling @ 75%: Repool db1094 after cloning another host', diff saved to https://phabricator.wikimedia.org/P14134 and previous config saved to /var/cache/conftool/dbconfig/20210202-142446-root.json [production]
14:21 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1002.eqiad.wmnet [production]