|
2019-05-30
§
|
| 18:49 |
<Urbanecm> |
Morning SWAT finished |
[production] |
| 18:47 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.7/extensions/GrowthExperiments/: [[:gerrit:513300|QuestionPoster: Correctly set timestamp when question is posted]] (T223338) (duration: 00m 51s) |
[production] |
| 18:26 |
<mutante> |
phab1003 - switch 'vcs' user to 'NP' to match phab1001 setup and then /srv/phab/phabricator# ./bin/config set diffusion.ssh-user vcs (T224677) |
[production] |
| 18:24 |
<XioNoX> |
bounce eqord-ulsfo interface to try to fix BFD sessions |
[production] |
| 18:12 |
<Krinkle> |
Running `php7adm /opcache-free` on mw1348 and mw1321, T224491 |
[production] |
| 18:12 |
<Krinkle> |
Running `php7adm /opcache-free` on mw1348 and mw1321 |
[production] |
| 18:11 |
<Krinkle> |
mw1348 (recent api/php72 100% experiment) shows signs of corruption |
[production] |
| 18:11 |
<Krinkle> |
mw1321 php7.2 shows signs of corruption for over 2 hours – https://phabricator.wikimedia.org/T224491#5224464 |
[production] |
| 18:03 |
<krinkle@deploy1001> |
Synchronized wmf-config/arclamp.php: (no justification provided) (duration: 00m 53s) |
[production] |
| 16:24 |
<bblack> |
re-pool cp3047 into service as ats-be - T222937 |
[production] |
| 16:04 |
<mutante> |
phab1001 - removing 2620:0:861:103:10:64:32:186/128 from eth0 |
[production] |
| 16:03 |
<mutante> |
phab1001 - removing 10.64.32.186/32 from eth0 |
[production] |
| 16:01 |
<mutante> |
phab1001 - removing git-ssh.wm.org IP from interface - phab1003 - activating IPv6 listen address for git-ssh |
[production] |
| 15:36 |
<jynus> |
stop es1019 for maintenance T213422 |
[production] |
| 15:26 |
<cmjohnson1> |
shutting down db1099 to swap DIMM T221502 |
[production] |
| 15:20 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1089 with full weight; depool es1019 (duration: 00m 52s) |
[production] |
| 15:19 |
<herron> |
performing rolling reboots of eqiad kafka main cluster hosts for security updates |
[production] |
| 15:06 |
<onimisionipe> |
pooled maps2004 - osm import is complete - T224395 |
[production] |
| 14:44 |
<andrewbogott> |
reimaging cloudvirtan1001 for T224566 |
[production] |
| 14:43 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:43 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:42 |
<andrewbogott> |
reimaging cloudvirtan1001 |
[production] |
| 14:29 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:29 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:25 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:25 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:22 |
<bblack> |
rebooting cp3047 (post-reimage/puppetization for T222937) |
[production] |
| 14:21 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:21 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:05 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:05 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:00 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:00 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:57 |
<jijiki> |
enable puppet on mw* in eqiad |
[production] |
| 13:44 |
<volans> |
rm /root/.ssh/known_hosts on cumin[12]001 |
[production] |
| 13:40 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:40 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:36 |
<zfilipin@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.7 |
[production] |
| 13:28 |
<jijiki> |
Enabling puppet on mw*.codfw.net |
[production] |
| 13:22 |
<zfilipin@deploy1001> |
Synchronized php-1.34.0-wmf.7/resources/src/jquery/jquery.suggestions.js: SWAT: [[gerrit:513237|jquery.suggestions: Do not show suggestions on prefilled values ([T224524])]] (duration: 00m 58s) |
[production] |
| 13:17 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1015.eqiad.wmnet |
[production] |
| 13:17 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1014.eqiad.wmnet |
[production] |
| 13:17 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1013.eqiad.wmnet |
[production] |
| 13:17 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1012.eqiad.wmnet |
[production] |
| 13:17 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1011.eqiad.wmnet |
[production] |
| 13:16 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1010.eqiad.wmnet |
[production] |
| 13:16 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1009.eqiad.wmnet |
[production] |
| 13:16 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1008.eqiad.wmnet |
[production] |
| 13:16 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1007.eqiad.wmnet |
[production] |
| 13:08 |
<bblack> |
cp3047 puppet-disable + depool for reimage to ATS - T222937 |
[production] |