|
2021-11-03
ยง
|
| 22:48 |
<ppchelko@deploy1002> |
Started deploy [restbase/deploy@664a2f8]: Add new wikis T292422 T294587 T294588 |
[production] |
| 22:47 |
<legoktm> |
upgraded scap on A:restbase (T294936) |
[production] |
| 22:38 |
<legoktm> |
upgrading scap on canaries (T294966) |
[production] |
| 22:34 |
<legoktm> |
upgraded apache2 on lists1001 |
[production] |
| 22:32 |
<legoktm> |
uploaded scap 4.0.3 to apt.wm.o for buster and stretch (T294966) |
[production] |
| 22:24 |
<twentyafterfour> |
restarted php7.3-fpm on phab1001 |
[production] |
| 22:24 |
<twentyafterfour> |
restarting phabricator to apply updates. |
[production] |
| 22:12 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wcqs2002.codfw.wmnet |
[production] |
| 22:12 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wcqs2001.codfw.wmnet |
[production] |
| 21:56 |
<ryankemper> |
T294961 [WCQS] Forcing recheck of `PyBal IPVS diff check` and `PyBal backends health check` |
[production] |
| 21:53 |
<ryankemper> |
T294961 [WCQS] Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/736564 and successfully ran `ryankemper@cumin1001:~$ sudo cumin 'A:icinga or A:dns-auth' run-puppet-agent` |
[production] |
| 21:47 |
<ryankemper> |
T294961 [WCQS] DNS changes rolled out, proceeding to the `lvs_setup` step: https://gerrit.wikimedia.org/r/c/operations/puppet/+/736564 |
[production] |
| 21:45 |
<ryankemper> |
T294961 [WCQS] Merged https://gerrit.wikimedia.org/r/c/operations/dns/+/736585, running `ryankemper@authdns1001:~$ sudo -i authdns-update` |
[production] |
| 21:38 |
<legoktm> |
upgrading/restarting apache2 on A:all-mw-eqiad |
[production] |
| 21:26 |
<legoktm> |
upgrading/restarting apache2 on A:all-mw-codfw |
[production] |
| 21:12 |
<legoktm> |
upgrading PHP 7.2 on labweb, deployment-servers |
[production] |
| 21:00 |
<legoktm> |
upgrading PHP 7.2 on A:snapshot |
[production] |
| 20:55 |
<legoktm> |
upgrading PHP 7.2 on A:parsoid |
[production] |
| 20:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:04 |
<eileen> |
civicrm revision changed from 93caef68ef to ac6f333db6, config revision is d3bb9999e7 |
[production] |
| 20:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:52 |
<dduvall@deploy1002> |
Synchronized php: group1 wikis to 1.38.0-wmf.7 refs T293948 (duration: 01m 03s) |
[production] |
| 19:51 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.7 refs T293948 |
[production] |
| 19:51 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:43 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wcqs2003.codfw.wmnet |
[production] |
| 19:42 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:35 |
<mutante> |
depooled wcqs2003 (pooled=inactive) because Icinga alerts that servers are down but pooled. not in production yet but issues (T294961) |
[production] |
| 19:33 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=wcqs2003.codfw.wmnet |
[production] |
| 19:33 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wcqs2003.codfw.wmnet |
[production] |
| 19:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:26 |
<mmandere> |
pool cp4035.ulsfo.wmnet - T290694 |
[production] |
| 19:19 |
<dduvall> |
1.38.0-wmf.7 now on group0. no new errors. leaving ~ 30 minutes before promoting group1 (T293948) |
[production] |
| 19:18 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:15 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.7 refs T293948 |
[production] |
| 19:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:10 |
<tgr> |
UTC evening deploys done |
[production] |
| 19:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:01 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:59 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 18:55 |
<razzi@cumin1001> |
START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 18:51 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4035.ulsfo.wmnet with OS buster |
[production] |
| 18:51 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:48 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:40 |
<legoktm> |
re-enabling puppet on lists1001 |
[production] |
| 18:38 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:34 |
<urbanecm> |
Purge https://en.wikipedia.org/.well-known/assetlinks.json, https://www.wikipedia.org/.well-known/assetlinks.json and https://wikipedia.org/.well-known/assetlinks.json (T294776) |
[production] |
| 18:34 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:24 |
<volans> |
rebooting ganeti-test2002 with fixed /etc/network/interfaces |
[production] |