|
2021-08-31
ยง
|
| 23:25 |
<dpifke@deploy1002> |
scap failed: average error rate on 3/6 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) |
[production] |
| 23:23 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:15 |
<mforns> |
failed deployment of refinery (v0.1.17) to an-test-coord1001.eqiad.wmnet (scap error) |
[production] |
| 23:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:14 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@a0f039b] (hadoop-test): Regular analytics weekly train TEST v0.1.17 [analytics/refinery@a0f039b] (duration: 13m 42s) |
[production] |
| 23:08 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 1437d99c1884c0695f02b81b724ec82a2bd3362e: Enable link recommendation frontent in dewiki and nlwiki (T288420, T285254) (duration: 01m 06s) |
[production] |
| 23:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:07 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 8997ae5d0b998839853aed2b246f5c88fe9d83eb: Fix wgDiscussionTools_sourcemodetoolbar settings (duration: 01m 22s) |
[production] |
| 23:01 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@a0f039b] (hadoop-test): Regular analytics weekly train TEST v0.1.17 [analytics/refinery@a0f039b] |
[production] |
| 23:00 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@a0f039b] (thin): Regular analytics weekly train THIN v0.1.17 [analytics/refinery@a0f039b] (duration: 00m 07s) |
[production] |
| 23:00 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@a0f039b] (thin): Regular analytics weekly train THIN v0.1.17 [analytics/refinery@a0f039b] |
[production] |
| 23:00 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@a0f039b]: Regular analytics weekly train v0.1.17 [analytics/refinery@a0f039b] (duration: 17m 39s) |
[production] |
| 22:42 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@a0f039b]: Regular analytics weekly train v0.1.17 [analytics/refinery@a0f039b] |
[production] |
| 21:58 |
<ejegg> |
switched Adyen to new Checkout integration |
[production] |
| 21:41 |
<dduvall@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 21:38 |
<dduvall@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 21:34 |
<dduvall@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 20:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:00 |
<twentyafterfour@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.21 refs T281161 |
[production] |
| 19:20 |
<brennen> |
gitlab1001: brief downtime for testing reconfiguration of cas3.session_duration |
[production] |
| 19:05 |
<twentyafterfour@deploy1002> |
Finished scap: testwikis wikis to 1.37.0-wmf.21 refs T281161 (duration: 35m 53s) |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:56 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:40 |
<ejegg> |
switched Adyen back to HPP integration |
[production] |
| 18:38 |
<ejegg> |
updated payments-wiki from 564daed816 to c4d56178d0, switched Adyen to Checkout integration |
[production] |
| 18:30 |
<twentyafterfour@deploy1002> |
Started scap: testwikis wikis to 1.37.0-wmf.21 refs T281161 |
[production] |
| 18:24 |
<twentyafterfour> |
ran `scap prep 1.37.0-wmf.21` and `scap apply-patches --train 1.37.0-wmf.21` refs T281162 |
[production] |
| 18:05 |
<XioNoX> |
re-pool eqsin-codfw link |
[production] |
| 16:18 |
<dcausse@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 16:14 |
<dcausse@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 16:08 |
<hnowlan@deploy1002> |
Finished deploy [restbase/deploy@09156c2]: fix core Title redirect loop (duration: 16m 02s) |
[production] |
| 15:52 |
<hnowlan@deploy1002> |
Started deploy [restbase/deploy@09156c2]: fix core Title redirect loop |
[production] |
| 14:29 |
<jbond> |
enable puppet fleet wide to post preform puppetdb maintance T263578 |
[production] |
| 14:29 |
<hashar> |
Restarting CI Jenkins for plugins upgrade |
[production] |
| 14:19 |
<ottomata> |
merged change to service_auto_restart.pp that changes the way service names are matched to be more explicit. tested in deployment prep and nothing bad happened. Logging in case something bad does happen in prod. https://gerrit.wikimedia.org/r/c/operations/puppet/+/697605 |
[production] |
| 14:09 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 14:09 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 14:07 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 14:05 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 14:05 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 14:03 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 14:03 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 14:02 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 14:02 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on puppetdb2002.codfw.wmnet with reason: puppetdb maintance - T289779 |
[production] |
| 14:02 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on puppetdb2002.codfw.wmnet with reason: puppetdb maintance - T289779 |
[production] |
| 14:02 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on puppetdb1002.eqiad.wmnet with reason: puppetdb maintance - T289779 |
[production] |
| 14:02 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on puppetdb1002.eqiad.wmnet with reason: puppetdb maintance - T289779 |
[production] |
| 14:01 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 14:00 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |