|
2019-07-11
§
|
| 10:28 |
<fsero> |
depooling ms-fe2005 for docker_registry_backups T227570 |
[production] |
| 10:08 |
<fsero> |
creating swift docker_registry_container_backup T227570 |
[production] |
| 09:56 |
<moritzm> |
re-enabling puppet (puppetdb reboots completed) |
[production] |
| 09:47 |
<moritzm> |
rebooting puppetdb1001 to pick up MDS-enabled qemu |
[production] |
| 09:35 |
<moritzm> |
rebooting puppetdb2001 to pick up MDS-enabled qemu |
[production] |
| 09:31 |
<moritzm> |
disabling puppet temporarily (for puppetdb reboots) |
[production] |
| 09:27 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:27 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:51 |
<godog> |
upload mtail 3.0.0~rc5-1~bpo9+1wmf1 to stretch-wikimedia - T225604 |
[production] |
| 08:14 |
<ema> |
cp-ulsfo: downgrade mtail to 3.0.0~rc5-1~bpo9+1 to fix varnishmtail-backend T225604 |
[production] |
| 07:43 |
<moritzm> |
installing ldap-codfw-replica* T227669 |
[production] |
| 07:31 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 07:21 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 07:21 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 07:11 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 07:10 |
<jmm@cumin2001> |
END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) |
[production] |
| 07:10 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 02:27 |
<ejegg> |
updated payments-wiki from 4c1261fe5d to 59ace50d66 |
[production] |
|
2019-07-10
§
|
| 23:16 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.13/extensions/CirrusSearch/includes: T227691 RedirectsAndIncomingLinks: succeede or fail, but not both (duration: 01m 02s) |
[production] |
| 23:02 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.13/extensions/OAuth/includes/backend/MWOAuthUtils.php: T227688 OAuth: Do not rely on array autocreation for custom User properties; re-try (duration: 00m 58s) |
[production] |
| 22:59 |
<jforrester@deploy1001> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 22:57 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.13/includes/user/User.php: T227688 User: support setting custom fields + array autocreation in non-existent field (duration: 00m 58s) |
[production] |
| 22:46 |
<shdubsh> |
downgrading cp4031 to mtail_3.0.0~rc5-1~bpo9+1wmf1 to fix varnishmtail T225604 |
[production] |
| 22:46 |
<jforrester@deploy1001> |
Synchronized w: T156319 Remove /w/skin-1.5 symlink (duration: 00m 58s) |
[production] |
| 22:16 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T212865 Stop configuring ZeroBanner and ZeroPortal, unused (duration: 00m 58s) |
[production] |
| 22:06 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T212865 Drop the ability to use ZeroBanner and ZeroPortal from production (duration: 00m 57s) |
[production] |
| 22:03 |
<jforrester@deploy1001> |
Synchronized wmf-config/mobile.php: T212865 Drop the ability to use ZeroBanner and ZeroPortal from production, mobile code (duration: 00m 57s) |
[production] |
| 21:59 |
<jforrester@deploy1001> |
Synchronized w/robots.php: T212865 Drop the special treatment for Wikipedia Zero (duration: 00m 58s) |
[production] |
| 21:55 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T212865 Drop the Wikipedia Zero debug log channel (duration: 00m 58s) |
[production] |
| 21:51 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T187716 Drop all zerowiki configuration (duration: 00m 58s) |
[production] |
| 21:50 |
<mutante> |
mwdebug1002 - php7adm /opcache-free because icinga showed a warning for opcache free space below 100MB |
[production] |
| 21:49 |
<jforrester@deploy1001> |
Synchronized dblists/: T187716 Mark zerowiki as deleted in dblists (duration: 01m 00s) |
[production] |
| 21:41 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T212865 Disable ZeroBanner on all wikis (duration: 00m 59s) |
[production] |
| 21:36 |
<mutante> |
mw1235 - restarting hhvm (socket timeout alert in icinga since about 1.5h) |
[production] |
| 21:35 |
<mutante> |
mw1290 - restarting hhvm (socket timeout alert in icinga since about 5h) |
[production] |
| 19:45 |
<hoo> |
Updated the Wikidata property suggester with data from the 2019-07-01 JSON dump and applied the T132839 workarounds |
[production] |
| 19:32 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Produce recentchange stream to eventgate-main - T211248 (duration: 00m 57s) |
[production] |
| 19:26 |
<otto@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Use wgEventServiceStreamConfig to configure wgRCFeeds eventbus. No-op in prod. - T211248 (duration: 00m 58s) |
[production] |
| 19:05 |
<jiji@deploy1001> |
Finished deploy [cpjobqueue/deploy@8761480]: Migrating rest of hightraffic jobs to PHP7 - T219150 (duration: 01m 00s) |
[production] |
| 19:04 |
<jiji@deploy1001> |
Started deploy [cpjobqueue/deploy@8761480]: Migrating rest of hightraffic jobs to PHP7 - T219150 |
[production] |
| 18:15 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.13/includes/Linker.php: T227656 Fix visibility of IPs that aren't suppressed (duration: 00m 59s) |
[production] |
| 17:54 |
<twentyafterfour> |
phabricator: hotfixing fatal error by pulling upstream fix ( see https://secure.phabricator.com/D20644 ) |
[production] |
| 16:09 |
<Urbanecm> |
Morning SWAT done |
[production] |
| 16:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[:gerrit:521865|Change bawikibooks logo to correct one according to community wish]] (2/2, T227418) (duration: 00m 58s) |
[production] |
| 16:07 |
<Urbanecm> |
Purged two urls for T227418 |
[production] |
| 16:06 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: SWAT: [[:gerrit:521865|Change bawikibooks logo to correct one according to community]] (1/2, T227418) (duration: 01m 16s) |
[production] |
| 16:04 |
<urbanecm@deploy1001> |
Synchronized dblists/commonsuploads.dblist: SWAT: [[:gerrit:521497|Disable local uploads on wuuwiki]] (T226764) (duration: 00m 58s) |
[production] |
| 15:23 |
<ema> |
cp-ulsfo: upgrade varnish to 5.1.3-1wm11 T227672 |
[production] |
| 15:08 |
<ema> |
restart wb2-phab wikibugs job |
[production] |
| 14:51 |
<ema> |
upload varnish 5.1.3-1wm11 to stretch-wikimedia T227672 |
[production] |