|
2021-09-27
ยง
|
| 18:59 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1075.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:57 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1074.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:56 |
<otto@deploy1002> |
Synchronized wmf-config/CommonSettings.php: REVERT: Enable x_client_ip_forwarding_enabled for eventgate-analytics and eventgate-analytics-external - T288853 (duration: 00m 56s) |
[production] |
| 18:55 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on elastic1073.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:53 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1074.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:53 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1073.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:52 |
<otto@deploy1002> |
scap failed: average error rate on 6/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) |
[production] |
| 18:50 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1072.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:48 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on elastic1069.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:47 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1072.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:46 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1069.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:43 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1071.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:42 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 18:41 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on elastic1070.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:41 |
<Amir1> |
Deployed patch for T284419 second time |
[production] |
| 18:40 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1071.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:39 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1070.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:35 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on elastic1068.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:33 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1068.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:32 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.1/includes/changetags/ChangeTags.php: b1f4b4e45b37792534e0aef4e636a26c369cc6a8: ChangeTags: Set interface flag when parsing tag names (T291776) (duration: 00m 56s) |
[production] |
| 18:30 |
<cmjohnson1> |
updating firmware on sessionstore1003 |
[production] |
| 18:21 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:16 |
<Amir1> |
Deployed patch for T284419 |
[production] |
| 18:10 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 2cb6f47ba4b739fa7bc8a2036b473c014ff69c8b: Growth: Promote 208 wikis out of dark mode (T290582) (duration: 00m 56s) |
[production] |
| 17:54 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 17:51 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 17:46 |
<ladsgroup@deploy1002> |
Synchronized php-1.38.0-wmf.1/includes/Title.php: Backport: [[gerrit:724109|Expand local URLs to absolute URLs in ParserOutput (T263581)]], Part IV (duration: 00m 56s) |
[production] |
| 17:44 |
<ladsgroup@deploy1002> |
Synchronized php-1.38.0-wmf.1/includes/parser/ParserCache.php: Backport: [[gerrit:724109|Expand local URLs to absolute URLs in ParserOutput (T263581)]], Part III (duration: 00m 56s) |
[production] |
| 17:43 |
<ladsgroup@deploy1002> |
Synchronized php-1.38.0-wmf.1/includes/parser/ParserOutput.php: Backport: [[gerrit:724109|Expand local URLs to absolute URLs in ParserOutput (T263581)]], Part II (duration: 00m 57s) |
[production] |
| 17:42 |
<ladsgroup@deploy1002> |
Synchronized php-1.38.0-wmf.1/includes/page/Article.php: Backport: [[gerrit:724109|Expand local URLs to absolute URLs in ParserOutput (T263581)]], Part I (duration: 00m 59s) |
[production] |
| 17:39 |
<volans> |
uploaded spicerack_1.0.2 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia |
[production] |
| 17:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 17:29 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 17:27 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=ldap-replica2006.wikimedia.org |
[production] |
| 17:26 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=ldap-replica2006.wikimedia.org |
[production] |
| 17:26 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=ldap-replica2005.wikimedia.org |
[production] |
| 17:26 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=ldap-replica200*.wikimedia.org |
[production] |
| 17:25 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=ldap-replica1004.wikimedia.org |
[production] |
| 17:25 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=ldap-replica1003.wikimedia.org |
[production] |
| 17:24 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=ldap-replica* |
[production] |
| 17:24 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=ldap-ro*.eqiad.wmnet |
[production] |
| 16:18 |
<otto@puppetmaster1001> |
conftool action : set/ttl=300; selector: dnsdisc=eventgate-logging-external |
[production] |
| 16:16 |
<otto@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw |
[production] |
| 16:14 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 16:14 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 16:12 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 16:10 |
<ottomata> |
reverting eventgate-logging-external chart change in codfw - T291504 |
[production] |
| 16:08 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ scap pull # T291836 |
[production] |
| 16:01 |
<urbanecm> |
Livehack debugging at mwmaint1002 for T291836 |
[production] |