2501-2550 of 10000 results (39ms)
2021-03-12 §
06:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1088 from dbctl T276025', diff saved to https://phabricator.wikimedia.org/P14803 and previous config saved to /var/cache/conftool/dbconfig/20210312-061118-marostegui.json [production]
04:14 <eileen> tools revision changed from d64b2f8cee to 532f8ecb33 [production]
01:30 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2215.codfw.wmnet [production]
00:58 <mutante> shutting down mw2215 [production]
00:57 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw2215.codfw.wmnet [production]
2021-03-11 §
22:55 <mutante> depooled mw2224 through mw2228 but not removing from DSH groups yet (T277119) [production]
22:55 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2228.codfw.wmnet [production]
22:55 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2227.codfw.wmnet [production]
22:55 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2226.codfw.wmnet [production]
22:54 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2225.codfw.wmnet [production]
22:54 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2224.codfw.wmnet [production]
22:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:48 <dzahn@cumin1001> START - Cookbook sre.dns.netbox [production]
22:47 <mutante> running DNS cookbook in an attempt to remove mw2216 [production]
22:47 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw2216.codfw.wmnet [production]
22:41 <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.34 [production]
22:36 <brennen> train status: 1.36.0-wmf.34 (T274938): T277229 and T266517 related issues hopefully resolved, rolling forward to all wikis [production]
22:34 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670879|Do not log script errors without file uri (T266517)]] (duration: 01m 07s) [production]
22:33 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:30 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/MobileFrontend/includes/: Backport: [[gerrit:670877|Revert "Fix: Save user options only once when Advanced Mode is toggled" (T277229)]] (duration: 01m 09s) [production]
22:28 <dzahn@cumin1001> START - Cookbook sre.dns.netbox [production]
21:57 <Amir1> run populate pages in cognate (T259360) [production]
21:28 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2222.codfw.wmnet [production]
21:27 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2223.codfw.wmnet [production]
21:27 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2221.codfw.wmnet [production]
21:27 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2220.codfw.wmnet [production]
21:21 <brennen@deploy1002> rebuilt and synchronized wikiversions files: Revert "all wikis to 1.36.0-wmf.34" [production]
21:20 <brennen> train status: 1.36.0-wmf.34 (T274938): rolling back to group1 and marking T277229 a train blocker [production]
21:17 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE [production]
21:15 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE [production]
21:14 <tgr@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:670858|Enable GrowthExperiments link recommendations on testwiki (T277173)] (duration: 00m 59s) [production]
21:13 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date (duration: 01m 53s) [production]
21:12 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date [production]
21:05 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw2216.codfw.wmnet [production]
21:04 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts mw2215.codfw.wmnet [production]
21:03 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
21:03 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
21:03 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw2215.codfw.wmnet [production]
21:00 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom [production]
21:00 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom [production]
21:00 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom [production]
21:00 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom [production]
21:00 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
21:00 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
20:58 <mutante> deactivating codfw API canaries on old hardware (T277119) [production]
20:57 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw2216.codfw.wmnet [production]
20:57 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw2215.codfw.wmnet [production]
20:50 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
20:46 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment (duration: 02m 09s) [production]
20:44 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment [production]