901-950 of 10000 results (30ms)
2021-03-25 ยง
17:45 <hnowlan> draining and restarting aqs1004-a cassandra [production]
17:16 <mbsantos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
17:14 <mbsantos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
17:08 <mbsantos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
16:39 <hashar> Restarted Apache 2 on contint2001 / contint1001 [production]
16:35 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
16:32 <moritzm> restarting apache on an-tool1007/turnilo [production]
16:27 <moritzm> restarting dnsdist/rdns-recursor on malmok [production]
16:24 <jbond42> restart slapd on ldap-replica [production]
16:22 <jbond42> restart slapd on ldap-corp [production]
16:20 <jbond42> restart apache on lists1002 [production]
16:18 <jbond42> restart apache on netbox [production]
16:13 <hashar@deploy1002> Synchronized php-1.36.0-wmf.36/extensions/ProofreadPage: Disallow negative or decimal values in pages tag - T278400 (duration: 01m 32s) [production]
16:12 <jbond42> restart routinator on rpki* [production]
16:12 <moritzm> restarting nginx on apt* [production]
16:10 <moritzm> restarting apache on dbmonitor [production]
16:08 <moritzm> restart Apacge on matomo/piwik [production]
16:03 <jbond42> restart apache service on gerrit [production]
16:02 <jbond42> restart idp service [production]
16:01 <ema> A:cp rolling ats-{tls,backend}-restart for openssl upgrades -- https://www.openssl.org/news/secadv/20210325.txt [production]
15:45 <moritzm> installing openssl updates on buster [production]
14:48 <herron@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:45 <herron@cumin1001> START - Cookbook sre.dns.netbox [production]
14:13 <twentyafterfour> update phabricator again (last night's update undid a hotfix that is now fixed properly) [production]
13:45 <moritzm> drain ganeti1009 [production]
13:27 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on webperf1001.eqiad.wmnet with reason: adapt RAM [production]
13:27 <jmm@cumin2001> START - Cookbook sre.hosts.downtime for 1:00:00 on webperf1001.eqiad.wmnet with reason: adapt RAM [production]
13:27 <moritzm> reduce webperf1001/webperf2001 to 4G RAM (xhgui has been split off to separate VMs) [production]
13:24 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1008.eqiad.wmnet [production]
13:18 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ganeti1008.eqiad.wmnet [production]
12:52 <hnowlan> aqs1004 nodetool-a cleanup finished [production]
12:14 <moritzm> drain ganeti1008 [production]
12:12 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1007.eqiad.wmnet [production]
12:08 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ganeti1007.eqiad.wmnet [production]
11:52 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:674861|Disable Legacy javascript in fawikiquote]] (T72470) (duration: 01m 07s) [production]
11:46 <moritzm> drain ganeti1007 [production]
11:44 <ladsgroup@deploy1002> Synchronized php-1.36.0-wmf.36/skins/Vector/resources: [[gerrit:674382|Inform anonymous A/B test by tracking time from navigationStart (T275807)]] (duration: 01m 09s) [production]
11:43 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1006.eqiad.wmnet [production]
11:39 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ganeti1006.eqiad.wmnet [production]
11:33 <ladsgroup@deploy1002> Synchronized dblists/: [[gerrit:674857|tawiki: Enable Growth features in dark mode]], Part II (T278369) (duration: 01m 07s) [production]
11:32 <ladsgroup@deploy1002> Synchronized wmf-config: [[gerrit:674857|tawiki: Enable Growth features in dark mode]] (T278369) (duration: 01m 30s) [production]
11:29 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pki2001.codfw.wmnet with reason: REIMAGE [production]
11:27 <jbond@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pki2001.codfw.wmnet with reason: REIMAGE [production]
11:24 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dns4001.wikimedia.org [production]
11:19 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pki1001.eqiad.wmnet with reason: REIMAGE [production]
11:18 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host dns4001.wikimedia.org [production]
11:17 <jbond@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pki1001.eqiad.wmnet with reason: REIMAGE [production]
11:10 <moritzm> drain ganeti1006 [production]
11:03 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1005.eqiad.wmnet [production]
10:58 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ganeti1005.eqiad.wmnet [production]