|
2021-01-29
§
|
| 23:26 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001 |
[production] |
| 22:36 |
<dancy@deploy1001> |
Finished scap: MW servers complaining about l10n files after .27 rollback (duration: 07m 22s) |
[production] |
| 22:29 |
<dancy@deploy1001> |
Started scap: MW servers complaining about l10n files after .27 rollback |
[production] |
| 22:26 |
<dancy@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.27 |
[production] |
| 22:20 |
<reedy@deploy1001> |
Synchronized php-1.36.0-wmf.27/includes/parser/CacheTime.php: CacheTime: Extra protection for rollback unserialization T273007 (duration: 01m 00s) |
[production] |
| 22:14 |
<dancy@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.28 |
[production] |
| 22:09 |
<dancy@deploy1001> |
scap failed: average error rate on 8/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) |
[production] |
| 21:42 |
<razzi> |
rebalance kafka partitions for codfw.resource_change |
[production] |
| 21:40 |
<razzi@cumin1001> |
START - Cookbook sre.kafka.reboot-workers for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001 |
[production] |
| 20:23 |
<razzi> |
rebalance kafka partitions for eventlogging_NavigationTiming |
[analytics] |
| 19:30 |
<razzi> |
rebalance kafka partitions for eqiad.mediawiki.revision-score |
[analytics] |
| 19:29 |
<razzi> |
rebalance kafka partitions for codfw.mediawiki.revision-score |
[analytics] |
| 19:26 |
<razzi@cumin1001> |
END (FAIL) - Cookbook sre.kafka.reboot-workers (exit_code=99) for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001 |
[production] |
| 19:26 |
<razzi@cumin1001> |
START - Cookbook sre.kafka.reboot-workers for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001 |
[production] |
| 19:14 |
<razzi> |
rebalance kafka partitions for eventlogging_CpuBenchmark |
[analytics] |
| 19:11 |
<razzi> |
rebalance kafka partitions for eqiad.mediawiki.page-links-change |
[analytics] |
| 19:10 |
<razzi> |
rebalance kafka partitions for codfw.mediawiki.page-links-change |
[analytics] |
| 18:51 |
<hashar> |
CI slightly overloaded due to a surge of library updates but is otherwise processing changes |
[releng] |
| 18:50 |
<hashar> |
CI slightly overloaded due to a surge of library updates but is otherwise processing changes |
[production] |
| 18:26 |
<Operator873> |
voiced ImprWikiImpr in #cvn-sw .. known as [[User:ImprovedWikiImprovment]] already. |
[cvn] |
| 18:21 |
<bstorm> |
deleting clouddb-toolsdb-03 as it isn't used |
[clouddb-services] |
| 17:31 |
<reedy@deploy1001> |
Synchronized php-1.36.0-wmf.28/extensions/WikiEditor/modules/jquery.wikiEditor.toolbar.config.js: T273231 (duration: 01m 02s) |
[production] |
| 16:56 |
<effie> |
depool mw1403 and mw1405 |
[production] |
| 15:46 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host an-test-presto1001.eqiad.wmnet |
[production] |
| 15:36 |
<andrewbogott> |
disabling puppet and some services on eqiad1 cloudcontrol nodes; replacing nova-placement-api with placement-api |
[admin] |
| 15:27 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host an-test-presto1001.eqiad.wmnet |
[production] |
| 14:58 |
<ariel@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on snapshot1007.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:56 |
<ariel@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on snapshot1007.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:33 |
<elukey> |
rollback presto upgrade, worker seems not able to announce themselves to the query coordinator |
[analytics] |
| 14:08 |
<elukey> |
upgrade presto to 0.246 (from 0.226) on an-presto1001 - worker node |
[analytics] |
| 14:02 |
<elukey> |
upgrade presto to 0.246 (from 0.226) on an-coord1001 - query coordinator |
[analytics] |
| 13:50 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 13:50 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 13:50 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 13:49 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 13:49 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 13:48 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 13:47 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 13:47 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 13:47 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 13:16 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 13:16 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 13:16 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 13:05 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 13:05 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 13:05 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 13:02 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 13:02 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |