|
2020-03-19
§
|
| 09:01 |
<akosiaris> |
restart recommendation-api on scb T246098 |
[production] |
| 09:00 |
<marostegui> |
Restart m2 primary database master - T246098 |
[production] |
| 08:48 |
<dcausse> |
depooling wdqs1006 to help catching up lag |
[production] |
| 08:43 |
<dcausse> |
restarting blazegraph on wdqs1006 (T242453) |
[production] |
| 07:54 |
<moritzm> |
installing cups updates from Stretch point release |
[production] |
| 07:48 |
<moritzm> |
installing libjaxen-java security updates from Stretch point release |
[production] |
| 07:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Update pc1008 spare situation T247787 (duration: 01m 09s) |
[production] |
| 06:49 |
<elukey> |
execute 'sudo rm /etc/logrotate.d/ceph-common' on cloudvirt-dev and cloudcontrol-dev to stop daily cronspam |
[production] |
| 06:46 |
<marostegui> |
Deploy schema change on testcommonswiki.globalimagelinks (empty table) on the s4 master T243987 |
[production] |
| 06:33 |
<marostegui> |
Upgrade db1132 without restarting T246098 |
[production] |
| 00:39 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikiws to 1.35.0-wmf.24 refs T233872 |
[production] |
| 00:31 |
<twentyafterfour@deploy1001> |
Synchronized php-1.35.0-wmf.24/skins/Vector/includes/templates/index.mustache: deploy https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/581116 which reverts https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/581054 refs T248010 (duration: 01m 07s) |
[production] |
| 00:18 |
<eileen> |
civicrm revision changed from a1b2cbeac1 to 1c477ff07f, config revision is 37232d8460 |
[production] |
|
2020-03-18
§
|
| 23:31 |
<twentyafterfour@deploy1001> |
Synchronized php-1.35.0-wmf.23/includes/TemplateParser.php: sync https://gerrit.wikimedia.org/r/c/mediawiki/core/+/581114/ refs T248010 (duration: 01m 07s) |
[production] |
| 23:26 |
<twentyafterfour@deploy1001> |
Synchronized php-1.35.0-wmf.24/includes/TemplateParser.php: sync https://gerrit.wikimedia.org/r/c/mediawiki/core/+/581115/ (duration: 01m 08s) |
[production] |
| 22:22 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:18 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:56 |
<Krinkle> |
krinkle@mw1385: scap pull # clean up AdHoc debugging for T248010 |
[production] |
| 21:16 |
<brennen@deploy1001> |
Synchronized php-1.35.0-wmf.24/skins/Vector/includes/templates/index.mustache: [[gerrit:581054|Change master template to force cache invalidation of partials]] (duration: 01m 06s) |
[production] |
| 21:11 |
<brennen@deploy1001> |
Synchronized php-1.35.0-wmf.23/skins/Vector/includes/templates/index.mustache: [[gerrit:581054|Change master template to force cache invalidation of partials]] (duration: 01m 15s) |
[production] |
| 20:04 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 19:58 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:49 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: Ensure fleet wide consistency |
[production] |
| 19:21 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 19:21 |
<mutante> |
shutting down (decom cookbook) elnath.codfw.wmnet (T188544) |
[production] |
| 19:20 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:15 |
<fdans@deploy1001> |
Finished deploy [analytics/refinery@549f6a4]: deploying analytics refinery (duration: 15m 02s) |
[production] |
| 19:11 |
<hashar> |
1.35.0-wmf.24 is on hold: too many blockers |
[production] |
| 19:00 |
<fdans@deploy1001> |
Started deploy [analytics/refinery@549f6a4]: deploying analytics refinery |
[production] |
| 18:32 |
<Lucas_WMDE> |
Morning SWAT done |
[production] |
| 18:30 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 18:27 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [[gerrit:579018|Update linter whitelist w/ parsoid11's IP address (T246833)]] (beta-only) (duration: 01m 04s) |
[production] |
| 18:20 |
<Lucas_WMDE> |
scap pull on mwdebug1001, attempting to fix mismatched wikiversions alert |
[production] |
| 18:14 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [[gerrit:580373|Add beta configuration for Wikibase reference formatting (T247416)]] (duration: 01m 08s) |
[production] |
| 18:13 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 18:13 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:580373|Add beta configuration for Wikibase reference formatting (T247416)]], take II (duration: 01m 07s) |
[production] |
| 18:11 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 18:11 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:580373|Add beta configuration for Wikibase reference formatting (T247416)]] (duration: 01m 07s) |
[production] |
| 16:43 |
<mutante> |
wtp1025 - Icinga alerted it's running out of disk - 'apt-get clean' lowered disk usage from 97% to 91% |
[production] |
| 16:00 |
<hashar@deploy1001> |
Finished scap: testwiki to 1.35.0-wmf.24 and rebuild l10n cache - T233872 (duration: 61m 23s) |
[production] |
| 14:58 |
<hashar@deploy1001> |
Started scap: testwiki to 1.35.0-wmf.24 and rebuild l10n cache - T233872 |
[production] |
| 14:41 |
<vgutierrez> |
disable TLS session tickets in ulsfo - T245616 T170567 |
[production] |
| 14:29 |
<godog> |
add debug to icinga2001 - T247538 |
[production] |
| 14:28 |
<_joe_> |
restarted php-fpm on mw1283, was throwing SIGILL |
[production] |
| 14:17 |
<marostegui> |
Rename wb_terms on codfw hosts: s8 (wikidatawiki - db2081), s3 (testwikidatawiki - db2109), s4 (commonswiki, testcommonswiki - db2106) T208425 |
[production] |
| 14:06 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.23 |
[production] |
| 11:59 |
<hashar@deploy1001> |
Synchronized php-1.35.0-wmf.24/includes/objectcache/ObjectCache.php: objectcache: Restore keyspace for LocalServerCache service - T247562 (duration: 01m 07s) |
[production] |
| 11:57 |
<hashar@deploy1001> |
Synchronized php-1.35.0-wmf.23/includes/objectcache/ObjectCache.php: objectcache: Restore keyspace for LocalServerCache service - T247562 (duration: 01m 10s) |
[production] |
| 11:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Decrease db1087, vslow host weight in main, given that the CPU across s8 is now doing a lot better', diff saved to https://phabricator.wikimedia.org/P10715 and previous config saved to /var/cache/conftool/dbconfig/20200318-114259-marostegui.json |
[production] |
| 11:17 |
<ema> |
upload atskafka 0.3 to buster-wikimedia T237993 |
[production] |