451-500 of 10000 results (17ms)
2020-10-02 §
07:29 <godog> prometheus codfw/k8s, add 50G to the LV [production]
07:23 <moritzm> installing libx11 security updates on buster [production]
06:51 <_joe_> restarting php-fpm on all appservers in eqiad, in batches of 10%, for testing the procedure suggested at T264362 [production]
05:48 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
05:43 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
05:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es2011 from dbctl T264261', diff saved to https://phabricator.wikimedia.org/P12893 and previous config saved to /var/cache/conftool/dbconfig/20201002-053020-marostegui.json [production]
2020-10-01 §
23:38 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% (duration: 00m 34s) [production]
23:38 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% [production]
23:33 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
23:15 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% (duration: 00m 24s) [production]
23:15 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% [production]
23:07 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
22:36 <James_F> Manually created mediawiki/extensions.git REL1_35 at 7ab9a74c9ebbb22ad9fb9b7c95c91b7fad8bf8c6 for T264365 [production]
22:35 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
22:23 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
22:09 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
22:03 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
22:00 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:58 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:29 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: rollback group0 as well T264363 [production]
21:29 <James_F> Manually created mediawiki/skins.git REL1_35 at 796693cb7a2ee3191fcbe19769d341bd0530bd4a for T264365 [production]
21:28 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:26 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:26 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: rollback group1 [production]
20:48 <twentyafterfour@deploy1001> Synchronized php: group1 wikis to 1.36.0-wmf.11 refs T263177 (duration: 01m 06s) [production]
20:47 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.11 refs T263177 [production]
20:19 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.11 [production]
20:08 <twentyafterfour@deploy1001> Synchronized php-1.36.0-wmf.11/includes/parser/: sync ParserCache patches to unblock the train T264257 T263177 (duration: 00m 59s) [production]
18:40 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cirrus: increase more_like recommendation cache from one to three days T264053 (duration: 00m 59s) [production]
17:49 <fdans@deploy1001> Finished deploy [analytics/refinery@530b339]: Regular analytics weekly train 530b339 (duration: 13m 42s) [production]
17:35 <fdans@deploy1001> Started deploy [analytics/refinery@530b339]: Regular analytics weekly train 530b339 [production]
17:26 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:24 <fdans@deploy1001> Finished deploy [analytics/refinery@530b339]: Regular analytics weekly train 530b339 (duration: 01m 34s) [production]
17:24 <mutante> etherpad1002 - attempted to upgrade Etherpad to newer version but wasn't working, reverted to previous one [production]
17:22 <fdans@deploy1001> Started deploy [analytics/refinery@530b339]: Regular analytics weekly train 530b339 [production]
17:16 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:46 <volans> migrating esams DNS records to the autogenerated ones from Netbox - T258729 [production]
16:19 <bblack> rebooting lvs1016 to a fresh state for interface config and error counters, etc - T264227 [production]
15:56 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:54 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:53 <bblack> lvs1016: re-disabled puppet with ticket ref in comment, downed interface enp5s0f0 since it's flapping furiously - T264227 [production]
15:53 <bblack> lvs1016: re-disabled puppet with ticket ref in comment, downed interface enp5s0f0 since it's flapping furiously [production]
14:55 <jayme> running ipvsadm -D -t 10.2.2.10:8081; ipvsadm -D -t 10.2.2.47:8889 on lvs1015.eqiad.wmnet - T244843 T255878 [production]
14:55 <moritzm> installing npm security updates on buster [production]
14:54 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:53 <jayme> running ipvsadm -D -t 10.2.1.10:8081; ipvsadm -D -t 10.2.1.47:8889 on lvs2010.codfw.wmnet,lvs2009.codfw.wmnet - T244843 T255878 [production]
14:52 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:50 <jayme> restarting pybal on lvs1015.eqiad.wmnet,lvs2009.codfw.wmnet - T244843 T255878 [production]
14:48 <jayme> restarting pybal on lvs2010.codfw.wmnet - T244843 T255878 [production]
14:42 <jayme> running puppet on lvs servers - T244843 T255878 [production]