2501-2550 of 10000 results (35ms)
2021-02-01 §
06:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1094 (re)pooling @ 50%: After fixing replication', diff saved to https://phabricator.wikimedia.org/P14062 and previous config saved to /var/cache/conftool/dbconfig/20210201-063422-root.json [production]
06:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1175 to dbctl, depooled T258361', diff saved to https://phabricator.wikimedia.org/P14061 and previous config saved to /var/cache/conftool/dbconfig/20210201-062358-marostegui.json [production]
06:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1094 (re)pooling @ 25%: After fixing replication', diff saved to https://phabricator.wikimedia.org/P14060 and previous config saved to /var/cache/conftool/dbconfig/20210201-061919-root.json [production]
06:10 <marostegui> Upgrade db2071 and db2102 to 10.4.18 [production]
06:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1094 (re)pooling @ 10%: After fixing replication', diff saved to https://phabricator.wikimedia.org/P14059 and previous config saved to /var/cache/conftool/dbconfig/20210201-060415-root.json [production]
05:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1094', diff saved to https://phabricator.wikimedia.org/P14058 and previous config saved to /var/cache/conftool/dbconfig/20210201-055851-marostegui.json [production]
2021-01-29 §
23:26 <razzi@cumin1001> END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001 [production]
22:36 <dancy@deploy1001> Finished scap: MW servers complaining about l10n files after .27 rollback (duration: 07m 22s) [production]
22:29 <dancy@deploy1001> Started scap: MW servers complaining about l10n files after .27 rollback [production]
22:26 <dancy@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.27 [production]
22:20 <reedy@deploy1001> Synchronized php-1.36.0-wmf.27/includes/parser/CacheTime.php: CacheTime: Extra protection for rollback unserialization T273007 (duration: 01m 00s) [production]
22:14 <dancy@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.28 [production]
22:09 <dancy@deploy1001> scap failed: average error rate on 8/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) [production]
21:42 <razzi> rebalance kafka partitions for codfw.resource_change [production]
21:40 <razzi@cumin1001> START - Cookbook sre.kafka.reboot-workers for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001 [production]
19:26 <razzi@cumin1001> END (FAIL) - Cookbook sre.kafka.reboot-workers (exit_code=99) for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001 [production]
19:26 <razzi@cumin1001> START - Cookbook sre.kafka.reboot-workers for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001 [production]
18:50 <hashar> CI slightly overloaded due to a surge of library updates but is otherwise processing changes [production]
17:31 <reedy@deploy1001> Synchronized php-1.36.0-wmf.28/extensions/WikiEditor/modules/jquery.wikiEditor.toolbar.config.js: T273231 (duration: 01m 02s) [production]
16:56 <effie> depool mw1403 and mw1405 [production]
15:46 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host an-test-presto1001.eqiad.wmnet [production]
15:27 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm for new host an-test-presto1001.eqiad.wmnet [production]
14:58 <ariel@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on snapshot1007.eqiad.wmnet with reason: REIMAGE [production]
14:56 <ariel@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on snapshot1007.eqiad.wmnet with reason: REIMAGE [production]
13:50 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
13:50 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
13:50 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
13:49 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
13:49 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
13:48 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
13:47 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
13:47 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
13:47 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
13:16 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
13:16 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
13:16 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
13:05 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
13:05 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
13:05 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
13:02 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
13:02 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
13:02 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
12:38 <hnowlan> uploaded osmborder_0.1.0-2~buster0 package to buster-wikimedia [production]
12:00 <gilles@deploy1001> Finished deploy [performance/coal@b0d3b59]: T271208 Filter out canary events (duration: 00m 06s) [production]
12:00 <gilles@deploy1001> Started deploy [performance/coal@b0d3b59]: T271208 Filter out canary events [production]
11:42 <dcausse@deploy1001> Synchronized wmf-config/unitConversionConfig.json: T270252: Update unitConversionConfig.json (duration: 01m 01s) [production]
11:39 <gilles@deploy1001> Finished deploy [performance/navtiming@ae8310a]: T271208 Fix canary event check (duration: 00m 05s) [production]
11:39 <gilles@deploy1001> Started deploy [performance/navtiming@ae8310a]: T271208 Fix canary event check [production]
11:26 <gilles@deploy1001> Finished deploy [performance/navtiming@e7712c3]: T271208 Log instead of hard error on missing wiki field (duration: 00m 06s) [production]
11:26 <gilles@deploy1001> Started deploy [performance/navtiming@e7712c3]: T271208 Log instead of hard error on missing wiki field [production]