|
2019-07-11
§
|
| 12:39 |
<jijiki> |
Disable puppet on mw1222, server will be depooled and pooled a few times for tests - T224538 |
[production] |
| 12:07 |
<godog> |
ms-be2031 raid controller firmware upgrade 4.52 -> 6.88 - T141756 |
[production] |
| 12:03 |
<godog> |
power reset ms-be2031, stuck and nothing on console |
[production] |
| 11:56 |
<Urbanecm> |
EU SWAT done |
[production] |
| 11:54 |
<urbanecm@deploy1001> |
Finished scap: Namespace translation for Punjabi (T226959) (duration: 30m 13s) |
[production] |
| 11:24 |
<urbanecm@deploy1001> |
Started scap: Namespace translation for Punjabi (T226959) |
[production] |
| 11:22 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[:gerrit:521933|Remove usergroup communityapps from officewiki]] (T227680) (duration: 01m 02s) |
[production] |
| 11:19 |
<urbanecm@deploy1001> |
Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: [[:gerrit:522035|Remove commonswiki from mobilemainpagelegacy]] (T227719) (duration: 00m 58s) |
[production] |
| 11:14 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [cirrus] Enable UTR30 as a lookup method for ns prefixes on group2 (duration: 01m 02s) |
[production] |
| 10:45 |
<moritzm> |
installing ldap-codfw-replica* |
[production] |
| 10:28 |
<fsero> |
depooling ms-fe2005 for docker_registry_backups T227570 |
[production] |
| 10:08 |
<fsero> |
creating swift docker_registry_container_backup T227570 |
[production] |
| 09:56 |
<moritzm> |
re-enabling puppet (puppetdb reboots completed) |
[production] |
| 09:47 |
<moritzm> |
rebooting puppetdb1001 to pick up MDS-enabled qemu |
[production] |
| 09:35 |
<moritzm> |
rebooting puppetdb2001 to pick up MDS-enabled qemu |
[production] |
| 09:31 |
<moritzm> |
disabling puppet temporarily (for puppetdb reboots) |
[production] |
| 09:27 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:27 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:51 |
<godog> |
upload mtail 3.0.0~rc5-1~bpo9+1wmf1 to stretch-wikimedia - T225604 |
[production] |
| 08:14 |
<ema> |
cp-ulsfo: downgrade mtail to 3.0.0~rc5-1~bpo9+1 to fix varnishmtail-backend T225604 |
[production] |
| 07:43 |
<moritzm> |
installing ldap-codfw-replica* T227669 |
[production] |
| 07:31 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 07:21 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 07:21 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 07:11 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 07:10 |
<jmm@cumin2001> |
END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) |
[production] |
| 07:10 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 02:27 |
<ejegg> |
updated payments-wiki from 4c1261fe5d to 59ace50d66 |
[production] |
|
2019-07-10
§
|
| 23:16 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.13/extensions/CirrusSearch/includes: T227691 RedirectsAndIncomingLinks: succeede or fail, but not both (duration: 01m 02s) |
[production] |
| 23:02 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.13/extensions/OAuth/includes/backend/MWOAuthUtils.php: T227688 OAuth: Do not rely on array autocreation for custom User properties; re-try (duration: 00m 58s) |
[production] |
| 22:59 |
<jforrester@deploy1001> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 22:57 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.13/includes/user/User.php: T227688 User: support setting custom fields + array autocreation in non-existent field (duration: 00m 58s) |
[production] |
| 22:46 |
<shdubsh> |
downgrading cp4031 to mtail_3.0.0~rc5-1~bpo9+1wmf1 to fix varnishmtail T225604 |
[production] |
| 22:46 |
<jforrester@deploy1001> |
Synchronized w: T156319 Remove /w/skin-1.5 symlink (duration: 00m 58s) |
[production] |
| 22:16 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T212865 Stop configuring ZeroBanner and ZeroPortal, unused (duration: 00m 58s) |
[production] |
| 22:06 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T212865 Drop the ability to use ZeroBanner and ZeroPortal from production (duration: 00m 57s) |
[production] |
| 22:03 |
<jforrester@deploy1001> |
Synchronized wmf-config/mobile.php: T212865 Drop the ability to use ZeroBanner and ZeroPortal from production, mobile code (duration: 00m 57s) |
[production] |
| 21:59 |
<jforrester@deploy1001> |
Synchronized w/robots.php: T212865 Drop the special treatment for Wikipedia Zero (duration: 00m 58s) |
[production] |
| 21:55 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T212865 Drop the Wikipedia Zero debug log channel (duration: 00m 58s) |
[production] |
| 21:51 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T187716 Drop all zerowiki configuration (duration: 00m 58s) |
[production] |
| 21:50 |
<mutante> |
mwdebug1002 - php7adm /opcache-free because icinga showed a warning for opcache free space below 100MB |
[production] |
| 21:49 |
<jforrester@deploy1001> |
Synchronized dblists/: T187716 Mark zerowiki as deleted in dblists (duration: 01m 00s) |
[production] |
| 21:41 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T212865 Disable ZeroBanner on all wikis (duration: 00m 59s) |
[production] |
| 21:36 |
<mutante> |
mw1235 - restarting hhvm (socket timeout alert in icinga since about 1.5h) |
[production] |
| 21:35 |
<mutante> |
mw1290 - restarting hhvm (socket timeout alert in icinga since about 5h) |
[production] |
| 19:45 |
<hoo> |
Updated the Wikidata property suggester with data from the 2019-07-01 JSON dump and applied the T132839 workarounds |
[production] |
| 19:32 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Produce recentchange stream to eventgate-main - T211248 (duration: 00m 57s) |
[production] |
| 19:26 |
<otto@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Use wgEventServiceStreamConfig to configure wgRCFeeds eventbus. No-op in prod. - T211248 (duration: 00m 58s) |
[production] |
| 19:05 |
<jiji@deploy1001> |
Finished deploy [cpjobqueue/deploy@8761480]: Migrating rest of hightraffic jobs to PHP7 - T219150 (duration: 01m 00s) |
[production] |
| 19:04 |
<jiji@deploy1001> |
Started deploy [cpjobqueue/deploy@8761480]: Migrating rest of hightraffic jobs to PHP7 - T219150 |
[production] |