|
2019-07-11
ยง
|
| 15:53 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:50 |
<XioNoX> |
deactivate ping-offload in codfw for server reboot |
[production] |
| 15:45 |
<thcipriani@deploy1001> |
Finished deploy [gerrit/gerrit@4daa16c]: it-phabricator plugin update (cobalt) (duration: 00m 11s) |
[production] |
| 15:45 |
<thcipriani@deploy1001> |
Started deploy [gerrit/gerrit@4daa16c]: it-phabricator plugin update (cobalt) |
[production] |
| 15:44 |
<thcipriani@deploy1001> |
Finished deploy [gerrit/gerrit@4daa16c]: it-phabricator plugin update (gerrit2001 only) (duration: 00m 11s) |
[production] |
| 15:44 |
<thcipriani@deploy1001> |
Started deploy [gerrit/gerrit@4daa16c]: it-phabricator plugin update (gerrit2001 only) |
[production] |
| 15:28 |
<gehel> |
setting CPU governor to performance for wdqs1004 - T225713 |
[production] |
| 15:28 |
<cdanis> |
upgrade to python3-conftool 1.1.0-1 on cp4022 |
[production] |
| 15:05 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo debdeploy deploy -u T197126-2019-07-11-conftool.yaml -s cp-canary |
[production] |
| 15:00 |
<hashar_> |
restarted Jenkins for plugins upgrades |
[production] |
| 14:57 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo debdeploy deploy -u T197126-2019-07-11-conftool.yaml -s mw-canary |
[production] |
| 14:55 |
<gehel> |
setting CPU governor to performance for elastic1052 - T225713 |
[production] |
| 14:51 |
<cdanis> |
upgrade to python3-conftool 1.1.0-1 on mwdebug2001 |
[production] |
| 14:45 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.13/includes/libs/rdbms/database/Database.php: 903f3f94f5d2e3 / T227708 (duration: 00m 59s) |
[production] |
| 14:26 |
<cdanis> |
cdanis@install1002.wikimedia.org ~ % sudo -E reprepro -C main include stretch-wikimedia /home/volans/conftool/stretch/conftool_1.1.0-1_amd64.changes |
[production] |
| 14:26 |
<cdanis> |
cdanis@install1002.wikimedia.org ~ % sudo -E reprepro -C main include jessie-wikimedia /home/volans/conftool/jessie/conftool_1.1.0-1+deb8u1_amd64.changes |
[production] |
| 14:26 |
<cdanis> |
cdanis@install1002.wikimedia.org ~ % sudo -E reprepro -C main include buster-wikimedia /home/volans/conftool/buster/conftool_1.1.0-1+deb10u1_amd64.changes |
[production] |
| 14:17 |
<ema> |
restart wikibugs |
[production] |
| 13:40 |
<godog> |
roll restart ms-be2016 ms-be2017 ms-be2018 ms-be2019 ms-be2020 ms-be2021 ms-be2028 ms-be2029 ms-be2030 ms-be2031 ms-be2032 ms-be2033 ms-be2034 ms-be2035 ms-be2036 - T225713 |
[production] |
| 13:00 |
<ema> |
cp-ulsfo: varnish frontend rolling restarts for 5.1.3-1wm11 upgrades T227672 |
[production] |
| 12:48 |
<ema> |
fleet-wide: remove obsolete file /etc/debdeploy-autorestarts.conf |
[production] |
| 12:44 |
<ema> |
cp-ulsfo: upgrade mtail to 3.0.0~rc5-1~bpo9+1wmf1 |
[production] |
| 12:44 |
<Urbanecm> |
Running purgePage.php on pages in Page: NS on pawikisource (T226959) |
[production] |
| 12:39 |
<jijiki> |
Disable puppet on mw1222, server will be depooled and pooled a few times for tests - T224538 |
[production] |
| 12:07 |
<godog> |
ms-be2031 raid controller firmware upgrade 4.52 -> 6.88 - T141756 |
[production] |
| 12:03 |
<godog> |
power reset ms-be2031, stuck and nothing on console |
[production] |
| 11:56 |
<Urbanecm> |
EU SWAT done |
[production] |
| 11:54 |
<urbanecm@deploy1001> |
Finished scap: Namespace translation for Punjabi (T226959) (duration: 30m 13s) |
[production] |
| 11:24 |
<urbanecm@deploy1001> |
Started scap: Namespace translation for Punjabi (T226959) |
[production] |
| 11:22 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[:gerrit:521933|Remove usergroup communityapps from officewiki]] (T227680) (duration: 01m 02s) |
[production] |
| 11:19 |
<urbanecm@deploy1001> |
Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: [[:gerrit:522035|Remove commonswiki from mobilemainpagelegacy]] (T227719) (duration: 00m 58s) |
[production] |
| 11:14 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [cirrus] Enable UTR30 as a lookup method for ns prefixes on group2 (duration: 01m 02s) |
[production] |
| 10:45 |
<moritzm> |
installing ldap-codfw-replica* |
[production] |
| 10:28 |
<fsero> |
depooling ms-fe2005 for docker_registry_backups T227570 |
[production] |
| 10:08 |
<fsero> |
creating swift docker_registry_container_backup T227570 |
[production] |
| 09:56 |
<moritzm> |
re-enabling puppet (puppetdb reboots completed) |
[production] |
| 09:47 |
<moritzm> |
rebooting puppetdb1001 to pick up MDS-enabled qemu |
[production] |
| 09:35 |
<moritzm> |
rebooting puppetdb2001 to pick up MDS-enabled qemu |
[production] |
| 09:31 |
<moritzm> |
disabling puppet temporarily (for puppetdb reboots) |
[production] |
| 09:27 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:27 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:51 |
<godog> |
upload mtail 3.0.0~rc5-1~bpo9+1wmf1 to stretch-wikimedia - T225604 |
[production] |
| 08:14 |
<ema> |
cp-ulsfo: downgrade mtail to 3.0.0~rc5-1~bpo9+1 to fix varnishmtail-backend T225604 |
[production] |
| 07:43 |
<moritzm> |
installing ldap-codfw-replica* T227669 |
[production] |
| 07:31 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 07:21 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 07:21 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 07:11 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 07:10 |
<jmm@cumin2001> |
END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) |
[production] |
| 07:10 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |