3651-3700 of 10000 results (36ms)
2020-07-29 Β§
00:48 <ryankemper> sudo -E cumin -b 10 'A:wdqs-all' 'sudo run-puppet-agent' [production]
00:18 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
00:14 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-07-28 Β§
23:38 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
23:37 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cirrus: reduce mlr window size on enwiki (duration: 01m 05s) [production]
23:36 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
23:34 <ebernhardson@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: cirrus: reduce mlr window size on enwiki (duration: 01m 06s) [production]
23:31 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
23:29 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
23:04 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Remove unused setting $wgGEHomepageSuggestedEditsNewAccountInitiatedPercentage (no-op) (duration: 01m 06s) [production]
22:55 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=wtp1046.eqiad.wmnet [production]
22:19 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet [production]
21:27 <dancy@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
21:24 <dancy@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
21:17 <dancy@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
20:38 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:36 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:02 <eileen> process-control config revision is b6ece03513 [production]
19:50 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:48 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:48 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:48 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:25 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
19:25 <andrew@cumin1001> START - Cookbook sre.hosts.decommission [production]
19:24 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
19:24 <andrew@cumin1001> START - Cookbook sre.hosts.decommission [production]
19:24 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
19:23 <andrew@cumin1001> START - Cookbook sre.hosts.decommission [production]
19:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1142', diff saved to https://phabricator.wikimedia.org/P12097 and previous config saved to /var/cache/conftool/dbconfig/20200728-191926-marostegui.json [production]
19:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1147', diff saved to https://phabricator.wikimedia.org/P12096 and previous config saved to /var/cache/conftool/dbconfig/20200728-191237-marostegui.json [production]
19:11 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@69bbbbb]: airflow: drop_old_data_daily: top_queries table renamed to fulltext_head_queries (duration: 00m 53s) [production]
19:11 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@69bbbbb]: airflow: drop_old_data_daily: top_queries table renamed to fulltext_head_queries [production]
19:09 <jhuneidi@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
19:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12095 and previous config saved to /var/cache/conftool/dbconfig/20200728-190933-marostegui.json [production]
19:06 <jhuneidi@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
19:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12094 and previous config saved to /var/cache/conftool/dbconfig/20200728-190517-marostegui.json [production]
19:03 <jhuneidi@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
19:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12093 and previous config saved to /var/cache/conftool/dbconfig/20200728-190137-marostegui.json [production]
18:35 <cdanis> βœ”οΈ cdanis@lvs1015.eqiad.wmnet ~ πŸ•β˜• sudo ipvsadm -D -t 10.2.2.51:9283 [production]
18:29 <cdanis> ❌cdanis@lvs1016.eqiad.wmnet ~ πŸ•β˜• sudo ipvsadm -D -t 10.2.2.51:9283 [production]
18:29 <catrope@deploy1001> Synchronized php-1.36.0-wmf.2/extensions/GrowthExperiments/extension.json: Fix reference to MentorChangeLogFormatter (T259041) (duration: 01m 05s) [production]
18:20 <catrope@deploy1001> Synchronized wmf-config/CommonSettings.php: No-op sync for wmgUseWikimediaApiPortal and wmgUseWikimediaApiPortalOAuth (2 of 2) (duration: 00m 58s) [production]
18:17 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: No-op sync for wmgUseWikimediaApiPortal and wmgUseWikimediaApiPortalOAuth (1 of 2) (duration: 01m 05s) [production]
18:16 <cdanis> primary pybal restart βœ”οΈ cdanis@lvs1015.eqiad.wmnet ~ πŸ•‘β˜• sudo systemctl restart pybal.service [production]
18:14 <cdanis> backup pybal restart: βœ”οΈ cdanis@lvs1016.eqiad.wmnet ~ πŸ•‘β˜• sudo systemctl restart pybal.service [production]
18:10 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:05 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
18:05 <catrope@deploy1001> Synchronized php-1.36.0-wmf.2/includes/libs/filebackend/SwiftFileBackend.php: Fix index error in SwiftFileBackend (T259023) (duration: 01m 07s) [production]
17:46 <volker-e@deploy1001> Finished deploy [design/style-guide@e3fda83]: Deploy design/style-guide: (duration: 00m 05s) [production]
17:46 <volker-e@deploy1001> Started deploy [design/style-guide@e3fda83]: Deploy design/style-guide: [production]