|
2020-07-29
§
|
| 04:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1142', diff saved to https://phabricator.wikimedia.org/P12100 and previous config saved to /var/cache/conftool/dbconfig/20200729-045859-marostegui.json |
[production] |
| 02:19 |
<tstarling@deploy1001> |
Synchronized wmf-config/CommonSettings.php: re-enable lilypond in safe mode (duration: 01m 09s) |
[production] |
| 01:47 |
<tstarling@deploy1001> |
Synchronized php-1.36.0-wmf.2/extensions/Score/includes/Score.php: work around firejail bug (duration: 01m 07s) |
[production] |
| 01:45 |
<tstarling@deploy1001> |
Synchronized php-1.36.0-wmf.1/extensions/Score/includes/Score.php: work around firejail bug (duration: 01m 08s) |
[production] |
| 01:19 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=wtp1048.eqiad.wmnet |
[production] |
| 01:15 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=wtp1047.eqiad.wmnet |
[production] |
| 00:53 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=wtp1046.eqiad.wmnet |
[production] |
| 00:48 |
<ryankemper> |
sudo -E cumin -b 10 'A:wdqs-all' 'sudo run-puppet-agent' |
[production] |
| 00:18 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:14 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
|
2020-07-28
§
|
| 23:38 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:37 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: cirrus: reduce mlr window size on enwiki (duration: 01m 05s) |
[production] |
| 23:36 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:34 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/CirrusSearch-common.php: cirrus: reduce mlr window size on enwiki (duration: 01m 06s) |
[production] |
| 23:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:29 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:04 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove unused setting $wgGEHomepageSuggestedEditsNewAccountInitiatedPercentage (no-op) (duration: 01m 06s) |
[production] |
| 22:55 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wtp1046.eqiad.wmnet |
[production] |
| 22:19 |
<jayme@cumin1001> |
conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet |
[production] |
| 21:27 |
<dancy@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 21:24 |
<dancy@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 21:17 |
<dancy@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 20:38 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:36 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:02 |
<eileen> |
process-control config revision is b6ece03513 |
[production] |
| 19:50 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 19:48 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:48 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:48 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:25 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 19:25 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:24 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 19:24 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:24 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 19:23 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1142', diff saved to https://phabricator.wikimedia.org/P12097 and previous config saved to /var/cache/conftool/dbconfig/20200728-191926-marostegui.json |
[production] |
| 19:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1147', diff saved to https://phabricator.wikimedia.org/P12096 and previous config saved to /var/cache/conftool/dbconfig/20200728-191237-marostegui.json |
[production] |
| 19:11 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@69bbbbb]: airflow: drop_old_data_daily: top_queries table renamed to fulltext_head_queries (duration: 00m 53s) |
[production] |
| 19:11 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@69bbbbb]: airflow: drop_old_data_daily: top_queries table renamed to fulltext_head_queries |
[production] |
| 19:09 |
<jhuneidi@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 19:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12095 and previous config saved to /var/cache/conftool/dbconfig/20200728-190933-marostegui.json |
[production] |
| 19:06 |
<jhuneidi@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 19:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12094 and previous config saved to /var/cache/conftool/dbconfig/20200728-190517-marostegui.json |
[production] |
| 19:03 |
<jhuneidi@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 19:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12093 and previous config saved to /var/cache/conftool/dbconfig/20200728-190137-marostegui.json |
[production] |
| 18:35 |
<cdanis> |
✔️ cdanis@lvs1015.eqiad.wmnet ~ 🕝☕ sudo ipvsadm -D -t 10.2.2.51:9283 |
[production] |
| 18:29 |
<cdanis> |
❌cdanis@lvs1016.eqiad.wmnet ~ 🕝☕ sudo ipvsadm -D -t 10.2.2.51:9283 |
[production] |
| 18:29 |
<catrope@deploy1001> |
Synchronized php-1.36.0-wmf.2/extensions/GrowthExperiments/extension.json: Fix reference to MentorChangeLogFormatter (T259041) (duration: 01m 05s) |
[production] |
| 18:20 |
<catrope@deploy1001> |
Synchronized wmf-config/CommonSettings.php: No-op sync for wmgUseWikimediaApiPortal and wmgUseWikimediaApiPortalOAuth (2 of 2) (duration: 00m 58s) |
[production] |
| 18:17 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: No-op sync for wmgUseWikimediaApiPortal and wmgUseWikimediaApiPortalOAuth (1 of 2) (duration: 01m 05s) |
[production] |