3651-3700 of 10000 results (85ms)
2020-03-06 §
01:34 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw230[1-9].codfw.wmnet [production]
01:34 <dzahn@cumin1001> conftool action : set/weight=15; selector: name=mw230[1-9].codfw.wmnet [production]
01:01 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
00:58 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
00:33 <cdanis> repool esams T246338 [production]
00:19 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
00:19 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
00:02 <cdanis> T246338 depool esams for router maintenance [production]
2020-03-05 §
23:55 <mutante> pooled mw2290 - noticed it was the only API appserver in codfw not pooled but did not see why, fine in Icinga and no open tickets/SAL [production]
23:55 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2290.codfw.wmnet [production]
23:30 <rzl@cumin1001> conftool action : set/pooled=yes; selector: name=mw1413.eqiad.wmnet [production]
23:27 <rzl@cumin1001> conftool action : set/weight=30; selector: name=mw1413.eqiad.wmnet [production]
23:26 <rlazarus> mw1413 test-reimage completed successfully, pooling [production]
23:03 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
23:01 <rzl@cumin1001> START - Cookbook sre.hosts.downtime [production]
22:50 <mutante> added 8 new appservers to pool in eqiad [production]
22:50 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw139[0-2].eqiad.wmnet [production]
22:47 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw138[5-9].eqiad.wmnet [production]
22:47 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw138[5-9].eqiad.wmnet [production]
22:46 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw139[0-2].eqiad.wmnet [production]
22:46 <dzahn@cumin1001> conftool action : set/weight=20; selector: name=mw139[0-2].eqiad.wmnet [production]
22:44 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw138[5-9]eqiad.wmnet [production]
22:42 <dzahn@cumin1001> conftool action : set/weight=20; selector: name=mw139[0-2]eqiad.wmnet [production]
22:41 <dzahn@cumin1001> conftool action : set/weight=20; selector: name=mw138[5-9]eqiad.wmnet [production]
22:41 <rlazarus> reimaging mw1413 (new appserver, not pooled) to test https://gerrit.wikimedia.org/r/c/576464 [production]
22:40 <mutante> [cumin1001:~] $ sudo -i cumin -b 15 'mw13[85-92].eqiad.wmnet' 'sudo -u dzahn scap pull' [production]
22:40 <rzl@cumin1001> conftool action : set/pooled=yes; selector: name=mw14(0[5-9]|1[0-2]).eqiad.wmnet [production]
22:40 <rzl@cumin1001> conftool action : set/weight=30; selector: name=mw14(0[5-9]|1[0-2]).eqiad.wmnet [production]
22:40 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
22:40 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
22:38 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 58s) [production]
22:36 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [ukwikinews] Add HD logos (duration: 00m 59s) [production]
22:35 <eileen> civicrm revision changed from 62e62e107c to 10506a9644, config revision is 734a7bfadd [production]
22:34 <jforrester@deploy1001> Synchronized static/images/project-logos/: [ukwikinews] Provide HD logos (duration: 00m 59s) [production]
22:27 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 56s) [production]
22:25 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [fawikivoyage] Add custom logos (duration: 00m 58s) [production]
22:22 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 59s) [production]
22:21 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Use HD logos at arwikibooks, cawikibooks, and plwikivoyage (duration: 00m 59s) [production]
22:17 <jforrester@deploy1001> Synchronized static/images/project-logos/: Provide HD logos for arwikibooks, cawikibooks, and plwikivoyage (duration: 01m 00s) [production]
22:14 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 59s) [production]
22:12 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Use HD logos at bnwikibooks, bnwikisource, and ukwikivoyage (duration: 00m 59s) [production]
22:10 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
22:09 <jforrester@deploy1001> Synchronized static/images/project-logos/: Provide HD logos for bnwikibooks, bnwikisource, and ukwikivoyage (duration: 01m 00s) [production]
22:07 <rzl@cumin1001> START - Cookbook sre.hosts.downtime [production]
22:05 <rzl@cumin1001> END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) [production]
22:05 <rzl@cumin1001> START - Cookbook sre.hosts.downtime [production]
22:01 <jforrester@deploy1001> Synchronized multiversion/MWConfigCacheGenerator.php: Stop loading four old logo dblists (duration: 00m 59s) [production]
21:40 <rzl@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:40 <rzl@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:39 <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings.php: MachineVision: Update label blacklist (once more for good measure) (duration: 00m 57s) [production]