|
2019-09-03
ยง
|
| 14:28 |
<hashar@deploy1001> |
Finished scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746 (duration: 50m 09s) |
[production] |
| 14:21 |
<moritzm> |
upgrading app server canaries to PHP 7.2.22 T230024 |
[production] |
| 13:44 |
<joal@deploy1001> |
Started deploy [analytics/refinery@8b17711]: Fixes for regualr analytics deploy |
[production] |
| 13:38 |
<hashar@deploy1001> |
Started scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746 |
[production] |
| 13:26 |
<hashar> |
Gerrit should be fine again, apparently was due to the wmf branch cut taking too much resources (sic) - T231872 filled to investigate |
[production] |
| 13:25 |
<hashar> |
1.34.0-wmf.21 cut |
[production] |
| 13:16 |
<hashar> |
Gerrit has some random times out from time to time (no reason) |
[production] |
| 13:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1073 from wikitech T229657', diff saved to https://phabricator.wikimedia.org/P9038 and previous config saved to /var/cache/conftool/dbconfig/20190903-131456-marostegui.json |
[production] |
| 13:13 |
<marostegui> |
Re-enable puppet on db1073 and db1133 T229657 |
[production] |
| 13:11 |
<marostegui> |
Reload haproxy on dbproxy1005 T229657 |
[production] |
| 13:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set wikitech back to RW after maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9037 and previous config saved to /var/cache/conftool/dbconfig/20190903-131000-marostegui.json |
[production] |
| 13:09 |
<moritzm> |
upgrading remaining mwdebug servers to PHP 7.2.22 T230024 |
[production] |
| 13:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1133 to wikitech master T229657', diff saved to https://phabricator.wikimedia.org/P9036 and previous config saved to /var/cache/conftool/dbconfig/20190903-130937-marostegui.json |
[production] |
| 13:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set wikitech as read-only for maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9035 and previous config saved to /var/cache/conftool/dbconfig/20190903-130839-marostegui.json |
[production] |
| 13:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set wikitech as read-only for maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9033 and previous config saved to /var/cache/conftool/dbconfig/20190903-130113-marostegui.json |
[production] |
| 13:00 |
<marostegui> |
Failover m5 from db1073 to db1133 - T229657 |
[production] |
| 12:52 |
<moritzm> |
uploaded PHP 7.2.22 to component/php72 T230024 |
[production] |
| 12:39 |
<moritzm> |
upgrading mwdebug2001 to PHP 7.2.22 |
[production] |
| 12:29 |
<hashar> |
Cutting wmf/1.34.0-wmf.21 # T220746 |
[production] |
| 12:19 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.20 |
[production] |
| 12:02 |
<marostegui> |
Disable puppet on db1073 and db1133 - T229657 |
[production] |
| 11:55 |
<marostegui> |
Change topology on m5 and make everything replicate from db1133 - T229657 |
[production] |
| 11:48 |
<marostegui> |
Downtime m5 hosts T229657 |
[production] |
| 11:35 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 1000 --sleep 2 (T225056) |
[production] |
| 11:29 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:29 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript namespaceDupes.php bswiki --fix (T231654) |
[production] |
| 11:28 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:533538|Fix wgMetaNamespaceTalk for bswiki (T231654)]] (duration: 00m 54s) |
[production] |
| 11:25 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:533930|Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016)]] (duration: 00m 52s) |
[production] |
| 11:10 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:533930|Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016)]] (duration: 00m 53s) |
[production] |
| 11:07 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:533882|Enable WRITE_BOTH for items term store for wikidatawiki (T225055)]] (duration: 00m 55s) |
[production] |
| 10:17 |
<ema> |
cp1083: varnish-backend-restart -- mbox lag, fetch failures |
[production] |
| 09:59 |
<_joe_> |
removing old lvs-related scripts from ores* |
[production] |
| 09:46 |
<moritzm> |
moved uid=smalyshev from cn=wmf to cn=nda |
[production] |
| 09:46 |
<mutante> |
install1002 - import GPG key for getenvoy repo, importing envoy for jessie with reprepro update |
[production] |
| 09:16 |
<hashar> |
Deploy refactor of Zuul pipelines which might mean that some repos/branches would miss jobs or have extra unwanted jobs. In such case please fill in a task against #continuous-integration-config |
[production] |
| 09:04 |
<ema> |
cp1085: varnish-backend-restart, mbox lag and fetch failures |
[production] |
| 09:03 |
<gehel> |
reset kartotherian password -T231842 |
[production] |
| 08:54 |
<ema> |
cp1089: varnish-backend-restart due to mbox lag and fetch failures |
[production] |
| 08:49 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be |
[production] |
| 08:49 |
<ema> |
cp1075: pool ats-be with caching enabled T228629 |
[production] |
| 08:26 |
<marostegui> |
Add REPLICATION grant to wikiuser and wikiadmin on db1073 with replication enabled - T229657 |
[production] |
| 08:21 |
<gehel> |
purging maps / info.json from cache - T231842 |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1133 with weight 0 T229657', diff saved to https://phabricator.wikimedia.org/P9031 and previous config saved to /var/cache/conftool/dbconfig/20190903-080958-marostegui.json |
[production] |
| 08:04 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@4810dfa]: Regular weekly analytics deploy train - Second try (duration: 00m 27s) |
[production] |
| 08:03 |
<joal@deploy1001> |
Started deploy [analytics/refinery@4810dfa]: Regular weekly analytics deploy train - Second try |
[production] |
| 08:02 |
<joal@deploy1001> |
deploy aborted: Regular weekly analytics deploy train (duration: 27m 47s) |
[production] |
| 07:16 |
<marostegui> |
Change min_replicas to 6 on s1 for eqiad and codfw T231019 |
[production] |
| 06:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1133 with weight 0 T229657', diff saved to https://phabricator.wikimedia.org/P9029 and previous config saved to /var/cache/conftool/dbconfig/20190903-063932-marostegui.json |
[production] |
| 06:10 |
<mutante> |
running puppet on cp-text_eqiad to switch people.wm.org to https backend |
[production] |
| 06:04 |
<marostegui> |
Change min_replicas to 4 on s7 for eqiad and codfw T231019 |
[production] |