|
2021-02-22
ยง
|
| 16:51 |
<Urbanecm> |
Run scap pull on mwmaint1002 to clear any local changes |
[production] |
| 16:50 |
<urbanecm@deploy1001> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 22s) |
[production] |
| 16:47 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Creating mniwiktionary (T273457) (duration: 00m 56s) |
[production] |
| 16:46 |
<urbanecm@deploy1001> |
rebuilt and synchronized wikiversions files: Creating mniwiktionary (T273457) |
[production] |
| 16:45 |
<urbanecm@deploy1001> |
Synchronized dblists: Creating mniwiktionary (T273457) (duration: 00m 56s) |
[production] |
| 16:44 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:44 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 16:44 |
<urbanecm@deploy1001> |
Synchronized wmf-config/db-codfw.php: Creating mniwiktionary (T273457) (duration: 00m 56s) |
[production] |
| 16:42 |
<urbanecm@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Creating mniwiktionary (T273457) (duration: 00m 55s) |
[production] |
| 16:37 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:36 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 16:26 |
<dpifke@deploy1001> |
Finished deploy [performance/arc-lamp@1f3bce1]: Deploy ArcLamp fixes for T273565 and T273640 (duration: 00m 05s) |
[production] |
| 16:26 |
<dpifke@deploy1001> |
Started deploy [performance/arc-lamp@1f3bce1]: Deploy ArcLamp fixes for T273565 and T273640 |
[production] |
| 16:19 |
<urbanecm@deploy1001> |
Synchronized langlist: Creating mniwiki (T273456) (duration: 00m 54s) |
[production] |
| 16:18 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Creating mniwiki (T273456) (duration: 00m 56s) |
[production] |
| 16:17 |
<urbanecm@deploy1001> |
Synchronized wmf-config/logos.php: Creating mniwiki (T273456) (duration: 00m 56s) |
[production] |
| 16:15 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: Creating mniwiki (T273456) (duration: 00m 55s) |
[production] |
| 16:14 |
<urbanecm@deploy1001> |
rebuilt and synchronized wikiversions files: Creating mniwiki (T273456) |
[production] |
| 16:13 |
<urbanecm@deploy1001> |
Synchronized dblists: Creating mniwiki (T273456) (duration: 00m 57s) |
[production] |
| 16:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/db-codfw.php: Creating mniwiki (T273456) (duration: 00m 55s) |
[production] |
| 16:11 |
<urbanecm@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Creating mniwiki (T273456) (duration: 00m 56s) |
[production] |
| 16:08 |
<urbanecm@deploy1001> |
Synchronized langlist: Creating altwiki (T271980) (duration: 00m 55s) |
[production] |
| 16:03 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Creating altwiki (T271980) (duration: 00m 55s) |
[production] |
| 16:01 |
<urbanecm@deploy1001> |
rebuilt and synchronized wikiversions files: Creating altwiki (T271980) |
[production] |
| 16:00 |
<urbanecm@deploy1001> |
Synchronized dblists: Creating altwiki (T271980) (duration: 00m 54s) |
[production] |
| 15:59 |
<urbanecm@deploy1001> |
Synchronized wmf-config/db-codfw.php: Creating altwiki (T271980) (duration: 00m 59s) |
[production] |
| 15:57 |
<Urbanecm> |
Temporarily replace /srv/mediawiki/php-1.36.0-wmf.31/extensions/WikimediaMaintenance/addWiki.php with /home/urbanecm/addWiki.php at mwmaint1002 to unbreak addWiki.php |
[production] |
| 15:53 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 15:43 |
<urbanecm@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Creating altwiki (T271980) (duration: 00m 56s) |
[production] |
| 15:32 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 14:16 |
<herron> |
roll restarting kafkamon hosts for updates |
[production] |
| 13:57 |
<filippo@cumin1001> |
conftool action : set/pooled=false; selector: dnsdisc=swift,name=codfw |
[production] |
| 13:47 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4001.ulsfo.wmnet |
[production] |
| 13:40 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.31/extensions/ContentTranslation/app/: f9e823e: CX3 Build 0.1.0+20210216 (fixes missing bits in T271397) (duration: 00m 55s) |
[production] |
| 13:40 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus3001.esams.wmnet |
[production] |
| 13:37 |
<moritzm> |
installing openldap security updates on corp replicas |
[production] |
| 13:36 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.31/extensions/FlaggedRevs/extension.json: a4cd98e7a581fe18634da05ba04eaf8035023c26: Grant sysops review and unreviewed pages right by default (apparently i forgot to rebase the first time, resync; T275293) (duration: 00m 57s) |
[production] |
| 13:32 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host prometheus4001.ulsfo.wmnet |
[production] |
| 13:31 |
<godog> |
reset-failed ifup@ens14 on prometheus3001 - T273026 |
[production] |
| 13:30 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cuminunpriv1001.eqiad.wmnet |
[production] |
| 13:29 |
<akosiaris> |
repool sessionstore in eqiad after sessionstore certificate refresh. T274564 |
[production] |
| 13:29 |
<akosiaris@cumin1001> |
conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=sessionstore |
[production] |
| 13:27 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host prometheus3001.esams.wmnet |
[production] |
| 13:26 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host cuminunpriv1001.eqiad.wmnet |
[production] |
| 13:17 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on cloudvirt-wdqs[1001-1003].eqiad.wmnet with reason: Restarting cloudcanary instances |
[production] |
| 13:16 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on cloudvirt-wdqs[1001-1003].eqiad.wmnet with reason: Restarting cloudcanary instances |
[production] |
| 13:16 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 27 hosts with reason: Restarting cloudcanary instances |
[production] |
| 13:16 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on 27 hosts with reason: Restarting cloudcanary instances |
[production] |
| 13:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Slowly repool db1175', diff saved to https://phabricator.wikimedia.org/P14439 and previous config saved to /var/cache/conftool/dbconfig/20210222-131153-root.json |
[production] |
| 12:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: Slowly repool db1175', diff saved to https://phabricator.wikimedia.org/P14438 and previous config saved to /var/cache/conftool/dbconfig/20210222-125650-root.json |
[production] |