3451-3500 of 10000 results (33ms)
2020-10-02 §
09:17 <jayme> restarting pybal on lvs1015.eqiad.wmnet,lvs2009.codfw.wmnet - T255875 T255869 [production]
09:14 <jayme> restarting pybal on lvs1016.eqiad.wmnet,lvs2010.codfw.wmnet - T255875 T255869 [production]
09:12 <jayme> running puppet on lvs servers - T255875 T255869 [production]
09:11 <arturo> added helm3 package to buster-wikimedia/thirdparty/kubeadm-k8s-1-17 (T264221) [production]
09:09 <jmm@cumin1001> START - Cookbook sre.ganeti.makevm [production]
09:08 <jmm@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
09:08 <jmm@cumin1001> START - Cookbook sre.ganeti.makevm [production]
09:07 <hnowlan> bootstrapping restbase1029-b cassandra [production]
09:05 <hashar> gerrit: running garbage collector [production]
09:00 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:00 <root@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:00 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:00 <root@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:59 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:59 <root@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:54 <dcausse@deploy1001> Finished deploy [wikimedia/discovery/analytics@5713fb0]: Test stat1007 deploy (duration: 00m 03s) [production]
08:54 <dcausse@deploy1001> Started deploy [wikimedia/discovery/analytics@5713fb0]: Test stat1007 deploy [production]
08:42 <dcausse@deploy1001> Finished deploy [wikimedia/discovery/analytics@5713fb0]: Test stat1007 deploy (duration: 00m 34s) [production]
08:41 <dcausse@deploy1001> Started deploy [wikimedia/discovery/analytics@5713fb0]: Test stat1007 deploy [production]
08:30 <dcausse@deploy1001> Finished deploy [wikimedia/discovery/analytics@5713fb0]: Fix lexeme dumps expected date (duration: 00m 33s) [production]
08:30 <dcausse@deploy1001> Started deploy [wikimedia/discovery/analytics@5713fb0]: Fix lexeme dumps expected date [production]
08:29 <moritzm> installing pyzmq bugfix update from buster point release [production]
08:24 <moritzm> installing nginx security updates on puppetdb* [production]
08:17 <dcausse@deploy1001> Finished deploy [wikimedia/discovery/analytics@5713fb0]: Fix lexeme dumps expected date (duration: 01m 35s) [production]
08:16 <dcausse@deploy1001> Started deploy [wikimedia/discovery/analytics@5713fb0]: Fix lexeme dumps expected date [production]
07:42 <moritzm> installing libcommons-compress-java security updates [production]
07:35 <godog> swift codfw-prod bump weight for ms-be2057 - T261633 [production]
07:29 <godog> prometheus codfw/k8s, add 50G to the LV [production]
07:23 <moritzm> installing libx11 security updates on buster [production]
06:51 <_joe_> restarting php-fpm on all appservers in eqiad, in batches of 10%, for testing the procedure suggested at T264362 [production]
05:48 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
05:43 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
05:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es2011 from dbctl T264261', diff saved to https://phabricator.wikimedia.org/P12893 and previous config saved to /var/cache/conftool/dbconfig/20201002-053020-marostegui.json [production]
2020-10-01 §
23:38 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% (duration: 00m 34s) [production]
23:38 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% [production]
23:33 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
23:15 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% (duration: 00m 24s) [production]
23:15 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% [production]
23:07 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
22:36 <James_F> Manually created mediawiki/extensions.git REL1_35 at 7ab9a74c9ebbb22ad9fb9b7c95c91b7fad8bf8c6 for T264365 [production]
22:35 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
22:23 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
22:09 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
22:03 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
22:00 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:58 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:29 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: rollback group0 as well T264363 [production]
21:29 <James_F> Manually created mediawiki/skins.git REL1_35 at 796693cb7a2ee3191fcbe19769d341bd0530bd4a for T264365 [production]
21:28 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:26 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]