2401-2450 of 10000 results (63ms)
2019-07-11 §
14:26 <cdanis> cdanis@install1002.wikimedia.org ~ % sudo -E reprepro -C main include buster-wikimedia /home/volans/conftool/buster/conftool_1.1.0-1+deb10u1_amd64.changes [production]
14:17 <ema> restart wikibugs [production]
13:40 <godog> roll restart ms-be2016 ms-be2017 ms-be2018 ms-be2019 ms-be2020 ms-be2021 ms-be2028 ms-be2029 ms-be2030 ms-be2031 ms-be2032 ms-be2033 ms-be2034 ms-be2035 ms-be2036 - T225713 [production]
13:00 <ema> cp-ulsfo: varnish frontend rolling restarts for 5.1.3-1wm11 upgrades T227672 [production]
12:48 <ema> fleet-wide: remove obsolete file /etc/debdeploy-autorestarts.conf [production]
12:44 <ema> cp-ulsfo: upgrade mtail to 3.0.0~rc5-1~bpo9+1wmf1 [production]
12:44 <Urbanecm> Running purgePage.php on pages in Page: NS on pawikisource (T226959) [production]
12:39 <jijiki> Disable puppet on mw1222, server will be depooled and pooled a few times for tests - T224538 [production]
12:07 <godog> ms-be2031 raid controller firmware upgrade 4.52 -> 6.88 - T141756 [production]
12:03 <godog> power reset ms-be2031, stuck and nothing on console [production]
11:56 <Urbanecm> EU SWAT done [production]
11:54 <urbanecm@deploy1001> Finished scap: Namespace translation for Punjabi (T226959) (duration: 30m 13s) [production]
11:24 <urbanecm@deploy1001> Started scap: Namespace translation for Punjabi (T226959) [production]
11:22 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[:gerrit:521933|Remove usergroup communityapps from officewiki]] (T227680) (duration: 01m 02s) [production]
11:19 <urbanecm@deploy1001> Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: [[:gerrit:522035|Remove commonswiki from mobilemainpagelegacy]] (T227719) (duration: 00m 58s) [production]
11:14 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [cirrus] Enable UTR30 as a lookup method for ns prefixes on group2 (duration: 01m 02s) [production]
10:45 <moritzm> installing ldap-codfw-replica* [production]
10:28 <fsero> depooling ms-fe2005 for docker_registry_backups T227570 [production]
10:08 <fsero> creating swift docker_registry_container_backup T227570 [production]
09:56 <moritzm> re-enabling puppet (puppetdb reboots completed) [production]
09:47 <moritzm> rebooting puppetdb1001 to pick up MDS-enabled qemu [production]
09:35 <moritzm> rebooting puppetdb2001 to pick up MDS-enabled qemu [production]
09:31 <moritzm> disabling puppet temporarily (for puppetdb reboots) [production]
09:27 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:27 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
08:51 <godog> upload mtail 3.0.0~rc5-1~bpo9+1wmf1 to stretch-wikimedia - T225604 [production]
08:14 <ema> cp-ulsfo: downgrade mtail to 3.0.0~rc5-1~bpo9+1 to fix varnishmtail-backend T225604 [production]
07:43 <moritzm> installing ldap-codfw-replica* T227669 [production]
07:31 <jmm@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
07:21 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
07:21 <jmm@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
07:11 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
07:10 <jmm@cumin2001> END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) [production]
07:10 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
02:27 <ejegg> updated payments-wiki from 4c1261fe5d to 59ace50d66 [production]
2019-07-10 §
23:16 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.13/extensions/CirrusSearch/includes: T227691 RedirectsAndIncomingLinks: succeede or fail, but not both (duration: 01m 02s) [production]
23:02 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.13/extensions/OAuth/includes/backend/MWOAuthUtils.php: T227688 OAuth: Do not rely on array autocreation for custom User properties; re-try (duration: 00m 58s) [production]
22:59 <jforrester@deploy1001> scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
22:57 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.13/includes/user/User.php: T227688 User: support setting custom fields + array autocreation in non-existent field (duration: 00m 58s) [production]
22:46 <shdubsh> downgrading cp4031 to mtail_3.0.0~rc5-1~bpo9+1wmf1 to fix varnishmtail T225604 [production]
22:46 <jforrester@deploy1001> Synchronized w: T156319 Remove /w/skin-1.5 symlink (duration: 00m 58s) [production]
22:16 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T212865 Stop configuring ZeroBanner and ZeroPortal, unused (duration: 00m 58s) [production]
22:06 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T212865 Drop the ability to use ZeroBanner and ZeroPortal from production (duration: 00m 57s) [production]
22:03 <jforrester@deploy1001> Synchronized wmf-config/mobile.php: T212865 Drop the ability to use ZeroBanner and ZeroPortal from production, mobile code (duration: 00m 57s) [production]
21:59 <jforrester@deploy1001> Synchronized w/robots.php: T212865 Drop the special treatment for Wikipedia Zero (duration: 00m 58s) [production]
21:55 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T212865 Drop the Wikipedia Zero debug log channel (duration: 00m 58s) [production]
21:51 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T187716 Drop all zerowiki configuration (duration: 00m 58s) [production]
21:50 <mutante> mwdebug1002 - php7adm /opcache-free because icinga showed a warning for opcache free space below 100MB [production]
21:49 <jforrester@deploy1001> Synchronized dblists/: T187716 Mark zerowiki as deleted in dblists (duration: 01m 00s) [production]
21:41 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T212865 Disable ZeroBanner on all wikis (duration: 00m 59s) [production]