|
2021-11-04
§
|
| 10:27 |
<mmandere> |
depool cp4036.ulsfo.wmnet - T290694 |
[production] |
| 10:21 |
<mmandere> |
pool cp4034.ulsfo.wmnet - T290694 |
[production] |
| 10:01 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4034.ulsfo.wmnet with OS buster |
[production] |
| 09:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17688 and previous config saved to /var/cache/conftool/dbconfig/20211104-093247-root.json |
[production] |
| 09:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17687 and previous config saved to /var/cache/conftool/dbconfig/20211104-091744-root.json |
[production] |
| 09:12 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp4034.ulsfo.wmnet with OS buster |
[production] |
| 09:09 |
<mmandere> |
depool cp4034.ulsfo.wmnet - T290694 |
[production] |
| 09:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17686 and previous config saved to /var/cache/conftool/dbconfig/20211104-090240-root.json |
[production] |
| 08:56 |
<dcausse> |
restarting blazegraph on wdqs1012 (stuck for the past 6 hours) |
[production] |
| 08:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17685 and previous config saved to /var/cache/conftool/dbconfig/20211104-084736-root.json |
[production] |
| 08:37 |
<_joe_> |
ipvsadm -Dt 10.2.2.67:443 on lvs101{5,6} |
[production] |
| 08:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17684 and previous config saved to /var/cache/conftool/dbconfig/20211104-083233-root.json |
[production] |
| 08:29 |
<_joe_> |
restarting pybal on low-traffic nodes in eqiad and codfw |
[production] |
| 08:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17683 and previous config saved to /var/cache/conftool/dbconfig/20211104-081729-root.json |
[production] |
| 08:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly pool db1163', diff saved to https://phabricator.wikimedia.org/P17682 and previous config saved to /var/cache/conftool/dbconfig/20211104-081726-marostegui.json |
[production] |
| 07:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17681 and previous config saved to /var/cache/conftool/dbconfig/20211104-074346-root.json |
[production] |
| 05:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight for the old special replicas T263127', diff saved to https://phabricator.wikimedia.org/P17679 and previous config saved to /var/cache/conftool/dbconfig/20211104-055419-marostegui.json |
[production] |
| 00:26 |
<tgr> |
UTC late deploys done |
[production] |
| 00:25 |
<tgr@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:736520|Add Wikivoyage in wgImportSources to enwikiversity (T294928)]] (duration: 01m 05s) |
[production] |
| 00:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:09 |
<tgr@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:736593|Enable GrowthExperiments image recommendations on ar,bn,cs,vi (T294878)]] (duration: 01m 03s) |
[production] |
| 00:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:01 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.6/extensions/GrowthExperiments: Backport: [[gerrit:736518|Add Image: add HTTP proxy config (T290949)]] [[gerrit:736519|Add Image: Harden API response parsing]] (duration: 01m 05s) |
[production] |
|
2021-11-03
§
|
| 23:57 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:54 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:22 |
<legoktm> |
reverted canaries back to scap 4.0.2 |
[production] |
| 23:20 |
<legoktm> |
uploaded scap 4.0.3-1+really4.0.2 to apt.wm.o for buster/stretch |
[production] |
| 23:02 |
<legoktm@deploy1002> |
Finished deploy [restbase/deploy@664a2f8]: (no justification provided) (duration: 00m 50s) |
[production] |
| 23:01 |
<legoktm@deploy1002> |
Started deploy [restbase/deploy@664a2f8]: (no justification provided) |
[production] |
| 22:48 |
<ppchelko@deploy1002> |
Finished deploy [restbase/deploy@664a2f8]: Add new wikis T292422 T294587 T294588 (duration: 00m 10s) |
[production] |
| 22:48 |
<ppchelko@deploy1002> |
Started deploy [restbase/deploy@664a2f8]: Add new wikis T292422 T294587 T294588 |
[production] |
| 22:47 |
<legoktm> |
upgraded scap on A:restbase (T294936) |
[production] |
| 22:38 |
<legoktm> |
upgrading scap on canaries (T294966) |
[production] |
| 22:34 |
<legoktm> |
upgraded apache2 on lists1001 |
[production] |
| 22:32 |
<legoktm> |
uploaded scap 4.0.3 to apt.wm.o for buster and stretch (T294966) |
[production] |
| 22:24 |
<twentyafterfour> |
restarted php7.3-fpm on phab1001 |
[production] |
| 22:24 |
<twentyafterfour> |
restarting phabricator to apply updates. |
[production] |
| 22:12 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wcqs2002.codfw.wmnet |
[production] |
| 22:12 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wcqs2001.codfw.wmnet |
[production] |
| 21:56 |
<ryankemper> |
T294961 [WCQS] Forcing recheck of `PyBal IPVS diff check` and `PyBal backends health check` |
[production] |
| 21:53 |
<ryankemper> |
T294961 [WCQS] Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/736564 and successfully ran `ryankemper@cumin1001:~$ sudo cumin 'A:icinga or A:dns-auth' run-puppet-agent` |
[production] |
| 21:47 |
<ryankemper> |
T294961 [WCQS] DNS changes rolled out, proceeding to the `lvs_setup` step: https://gerrit.wikimedia.org/r/c/operations/puppet/+/736564 |
[production] |
| 21:45 |
<ryankemper> |
T294961 [WCQS] Merged https://gerrit.wikimedia.org/r/c/operations/dns/+/736585, running `ryankemper@authdns1001:~$ sudo -i authdns-update` |
[production] |
| 21:38 |
<legoktm> |
upgrading/restarting apache2 on A:all-mw-eqiad |
[production] |
| 21:26 |
<legoktm> |
upgrading/restarting apache2 on A:all-mw-codfw |
[production] |
| 21:12 |
<legoktm> |
upgrading PHP 7.2 on labweb, deployment-servers |
[production] |
| 21:00 |
<legoktm> |
upgrading PHP 7.2 on A:snapshot |
[production] |
| 20:55 |
<legoktm> |
upgrading PHP 7.2 on A:parsoid |
[production] |