|
2021-03-11
§
|
| 07:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 30%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14753 and previous config saved to /var/cache/conftool/dbconfig/20210311-070734-root.json |
[production] |
| 06:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 10%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14752 and previous config saved to /var/cache/conftool/dbconfig/20210311-065230-root.json |
[production] |
| 06:48 |
<marostegui> |
Stop mysql on db2108 to clone db2148 T275633 |
[production] |
| 06:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2108 T275633', diff saved to https://phabricator.wikimedia.org/P14750 and previous config saved to /var/cache/conftool/dbconfig/20210311-064821-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P14749 and previous config saved to /var/cache/conftool/dbconfig/20210311-063814-marostegui.json |
[production] |
| 06:36 |
<marostegui> |
Drop testreduce from m5 - T276787 |
[production] |
| 05:34 |
<thcipriani> |
restarted apache2 on gerrit1001 |
[production] |
| 00:50 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2219.codfw.wmnet |
[production] |
| 00:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2218.codfw.wmnet |
[production] |
| 00:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2217.codfw.wmnet |
[production] |
| 00:48 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2216.codfw.wmnet |
[production] |
| 00:48 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2215.codfw.wmnet |
[production] |
|
2021-03-10
§
|
| 23:49 |
<mholloway-shell@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/EventLogging: EventLogging: Stream always in sample if the user is in debugMode (T276515) (duration: 01m 23s) |
[production] |
| 23:41 |
<dwisehaupt> |
disabled silverpop daily run in process-control until utf8mb4 conversion completes on frdev1001 |
[production] |
| 23:12 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:10 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:10 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1002.eqiad.wmnet |
[production] |
| 23:01 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts registry1002.eqiad.wmnet |
[production] |
| 22:55 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry[2001-2002].codfw.wmnet |
[production] |
| 22:51 |
<andrewbogott> |
updating puppet compiler facts to catch up with a new custom fact |
[production] |
| 22:44 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts registry[2001-2002].codfw.wmnet |
[production] |
| 22:40 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1001.eqiad.wmnet |
[production] |
| 22:32 |
<brennen@deploy1002> |
Synchronized php: group1 wikis to 1.36.0-wmf.34 (duration: 01m 30s) |
[production] |
| 22:30 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.34 |
[production] |
| 22:27 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts registry1001.eqiad.wmnet |
[production] |
| 22:26 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): T277094 believed resolved, promoting to group1. |
[production] |
| 22:25 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670535|Fix client error logging (T277094)]] (duration: 01m 09s) |
[production] |
| 21:53 |
<mutante> |
ferm/iptables docker NAT rules applied by puppet on releases servers after breaking out fules into their own profile class (T276869) |
[production] |
| 21:51 |
<dwisehaupt> |
upgraded mariadb and keeping replication stopped on frdb1002 to start the utf8mb4 table alters under a root screen session |
[production] |
| 21:43 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): client errors may still be missing for group0; continuing to hold for T277094 until we know what's broken. |
[production] |
| 21:40 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670533|Revert "Error in shouldLog logic drops most errors" (T277094)]] (duration: 01m 08s) |
[production] |
| 21:38 |
<dwisehaupt> |
stopping mysql replication on frdev1001 and starting utf8mb4 table alters under a root screen session |
[production] |
| 21:38 |
<dwisehaupt> |
stopping mysql replication on frdb1003 and starting utf8mb4 table alters under a root screen session |
[production] |
| 21:30 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): logstash client error board was set up incorrectly; reverting earlier patch for T277094 and will proceed to group1. |
[production] |
| 21:19 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: cdc47f3e35e815081f787def2d51f3fd337ecf6c: jawiki: Growth features: Add help panel links (T276830) (duration: 01m 08s) |
[production] |
| 21:16 |
<eileen> |
civicrm revision changed from b13e70d968 to 550be50105, config revision is 970b10b0b3 |
[production] |
| 21:13 |
<cdanis@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 21:00 |
<cdanis@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:57 |
<cdanis@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:56 |
<Urbanecm> |
Fixing wrong sync message: urbanecm@deploy1002 Synchronized dblists/growthexperiments.dblist f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 08s) |
[production] |
| 20:56 |
<Urbanecm> |
Fixing wrong sync message: urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 07s) |
[production] |
| 20:54 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) |
[production] |
| 20:53 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 07s) |
[production] |
| 20:50 |
<cdanis@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:48 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) |
[production] |
| 20:41 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670529|Error in shouldLog logic drops most errors (T277094)]] (duration: 01m 14s) |
[production] |
| 20:36 |
<cdanis@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 19:58 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): currently blocked at group0 as client error logging is broken (UBN ticket incoming), will hold for patch. |
[production] |
| 19:37 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: a130e9f2eab6dec12aec4380efdfd6bde1767aeb: Enable Growth features on eowiki in stealth mode (T276123) (duration: 01m 08s) |
[production] |
| 19:35 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: REIMAGE |
[production] |