3101-3150 of 10000 results (37ms)
2021-04-28 ยง
14:44 <moritzm> imported gitlab-ce 13.9.7-ce.0 to apt.wikimedia.org [production]
14:40 <milimetric@deploy1002> Finished deploy [analytics/refinery@559d98d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@559d98d] (duration: 04m 59s) [production]
14:35 <milimetric@deploy1002> Started deploy [analytics/refinery@559d98d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@559d98d] [production]
14:34 <milimetric@deploy1002> Finished deploy [analytics/refinery@559d98d] (thin): Regular analytics weekly train THIN [analytics/refinery@559d98d] (duration: 00m 06s) [production]
14:34 <milimetric@deploy1002> Started deploy [analytics/refinery@559d98d] (thin): Regular analytics weekly train THIN [analytics/refinery@559d98d] [production]
14:34 <milimetric@deploy1002> Finished deploy [analytics/refinery@559d98d]: Regular analytics weekly train [analytics/refinery@559d98d] (duration: 03m 07s) [production]
14:32 <moritzm> installing iproute2 updates from buster point release [production]
14:31 <milimetric@deploy1002> Started deploy [analytics/refinery@559d98d]: Regular analytics weekly train [analytics/refinery@559d98d] [production]
14:30 <milimetric@deploy1002> deploy aborted: - (duration: 00m 00s) [production]
14:30 <milimetric@deploy1002> Started deploy [analytics/refinery@559d98d]: - [production]
14:30 <milimetric@deploy1002> Finished deploy [analytics/refinery@559d98d]: Regular analytics weekly train [analytics/refinery@559d98d] (duration: 12m 31s) [production]
14:26 <moritzm> installing net-snmp updates from buster point release [production]
14:17 <milimetric@deploy1002> Started deploy [analytics/refinery@559d98d]: Regular analytics weekly train [analytics/refinery@559d98d] [production]
13:59 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE [production]
13:57 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE [production]
13:15 <jayme> restarting pybal on lvs5001,lvs4005,lvs2007 - T271573 [production]
13:14 <liw@deploy1002> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 3.17.0-wmf.1" [production]
13:10 <jayme> restarting pybal on lvs5002,lvs4006,lvs2008 - T271573 [production]
13:04 <liw@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.3 (duration: 01m 07s) [production]
13:03 <jmm@cumin2001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
13:03 <liw@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.3 [production]
13:02 <moritzm> upgrading deployment servers to PHP 7.4.32 [production]
12:55 <moritzm> upgrading snapshot hosts to PHP 7.4.32 [production]
12:48 <jayme> restarting pybal on lvs2009 - T271573 [production]
12:45 <moritzm> upgrading labweb to PHP 7.4.32 [production]
12:43 <jmm@cumin2001> START - Cookbook sre.cassandra.roll-restart [production]
12:42 <jayme> restarting pybal on lvs5003,lvs4007 - T271573 [production]
12:39 <jayme> restarting pybal on lvs2010 - T271573 [production]
12:36 <jmm@cumin2001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
12:28 <apergos> manually edited /srv/deployment/dumps/dumps-cache/config on snapshots1011,12,13 to change deploy1001 to deploy1002 (where did it get the old value from? these are new installs!) [production]
12:16 <moritzm> rolling restart of cassandra in restbase-dev to pick up Java security updates [production]
12:15 <jmm@cumin2001> START - Cookbook sre.cassandra.roll-restart [production]
12:15 <jmm@cumin2001> END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) [production]
12:15 <jmm@cumin2001> START - Cookbook sre.cassandra.roll-restart [production]
11:53 <jayme> switching SRV record _etcd._tcp to new etcd cluster (for codfw, eqsin, ulsfo) [production]
11:22 <Urbanecm> EU B&C window done [production]
11:20 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.3/extensions/Popups/: 8d0ae5e8fedefa911fc216bfc810d7a6169ea7e5: Separate reference preview settings in beta & non-beta (T281235) (duration: 01m 08s) [production]
11:16 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: ddbc378e41783356e28cd90bbefa08624ea2844c: Enable partial action blocks on testwiki (T280528) (duration: 01m 07s) [production]
11:05 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: REIMAGE [production]
11:03 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1001.eqiad.wmnet with reason: REIMAGE [production]
11:03 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: REIMAGE [production]
11:01 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1001.eqiad.wmnet with reason: REIMAGE [production]
10:44 <jbond42> updated the check-raid nrpe script to python3 [production]
09:40 <moritzm> restarting Tomcat on idp-test1001 to pick up Java security updates [production]
09:21 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3316 (re)pooling @ 100%: Repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P15618 and previous config saved to /var/cache/conftool/dbconfig/20210428-092103-root.json [production]
09:19 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host contint1001.wikimedia.org [production]
09:12 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host contint1001.wikimedia.org [production]
09:09 <moritzm> restarting jenkins* on releases to pick up Java security updates [production]
09:08 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host contint2001.wikimedia.org [production]
09:06 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3316 (re)pooling @ 75%: Repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P15617 and previous config saved to /var/cache/conftool/dbconfig/20210428-090559-root.json [production]