|
2020-06-25
§
|
| 23:37 |
<mutante> |
puppetmaster - signing certs and initial puppet run for logstash1030/logstash1031 - no prod role yet |
[production] |
| 22:52 |
<bstorm> |
created paws-k8s-worker-5/6/7 as x-large nodes to bring the cluster up to roughly the same capacity as the existing one using soft anti-affinity T211096 T253267 |
[paws] |
| 22:43 |
<bstorm> |
bumped quota up to 24 instances, 128 GB RAM and 56 cores T211096 |
[paws] |
| 22:25 |
<mutante> |
puppetmaster - signing certs and initial run for logstash2030/2031 - no prod role yet |
[production] |
| 22:02 |
<dpifke> |
Testing changes 607369 and 607370 in beta, should only affect deployment-webperf12. |
[releng] |
| 21:49 |
<zhuyifei1999_> |
re-enabling puppet on tools-sgebastion-09 T256426 |
[tools] |
| 21:39 |
<zhuyifei1999_> |
disabling puppet on tools-sgebastion-09 so I can play with mount settings T256426 |
[tools] |
| 21:24 |
<bstorm> |
hard rebooting tools-sgebastion-09 |
[tools] |
| 20:57 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 20:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 19:30 |
<dcausse> |
repooling wdqs1007.eqiad.wmnet |
[production] |
| 19:05 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.38 |
[production] |
| 18:58 |
<mutante> |
LDAP - added qchris to archiva-deployers (T256404) |
[production] |
| 17:37 |
<mutante> |
mwmaint1002 - restarted apache2 to add server_headers snippet for T255629 - but not working as expected yet |
[production] |
| 17:20 |
<elukey> |
move RU jobs/timers from an-launcher1001 to an-launcher1002 |
[analytics] |
| 16:40 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 16:39 |
<bstorm> |
deleted the deployhook from the in-progress new cluster for now just in case T211096 |
[paws] |
| 16:31 |
<sukhe@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:31 |
<sukhe@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:28 |
<krinkle@deploy1001> |
Synchronized wmf-config/logging.php: Ia6ef7617d378 (duration: 01m 02s) |
[production] |
| 16:20 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 16:16 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 16:16 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 16:16 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 16:15 |
<Krinkle> |
I've deleted a "saved object" visualisation in logstash called "Production Errors & Deployments" which seemed to be corrupt and redirect random logstash dashboards to a management page. Backed up at https://phabricator.wikimedia.org/P11666 (NDA) |
[production] |
| 16:15 |
<moritzm> |
installing libxml2 security updates |
[production] |
| 16:12 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 16:09 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 16:07 |
<elukey> |
move all timers but RU from an-launcher1001 to 1002 (puppet disabled on 1001, all timers completed) |
[analytics] |
| 16:06 |
<moritzm> |
installing 4.9.210-1+deb9u1~deb8u1 on jessie hosts (fixed kernel for recent cacheoutattack CPU leaks) |
[production] |
| 16:04 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 16:03 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:59 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 15:55 |
<krinkle@deploy1001> |
Synchronized wmf-config/logging.php: I4c519f88c613fc (duration: 01m 05s) |
[production] |
| 15:54 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:53 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:51 |
<vgutierrez> |
upgrade ATS in eqiad to version 8.0.8 |
[production] |
| 15:44 |
<bstorm> |
deployed a proof-of-concept paws-public setup in the new cluster T255997 |
[paws] |
| 15:42 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups (duration: 05m 09s) |
[production] |
| 15:37 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups |
[production] |
| 15:37 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups (duration: 03m 38s) |
[production] |
| 15:34 |
<Reedy> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/607818 |
[releng] |
| 15:33 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups |
[production] |
| 15:33 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@821e96b]: Only emit vary: accept-language for feeds when it matters T256358, more groups (duration: 03m 24s) |
[production] |
| 15:30 |
<vgutierrez> |
upgrade ATS in codfw to version 8.0.8 |
[production] |
| 15:30 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |