|
2021-07-29
§
|
| 12:55 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@1e31cc6]: Increase mirrored traffic to tegola |
[production] |
| 12:22 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ganeti2026.codfw.wmnet |
[production] |
| 12:21 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet |
[production] |
| 10:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2104.codfw.wmnet with reason: REIMAGE |
[production] |
| 10:47 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2104.codfw.wmnet with reason: REIMAGE |
[production] |
| 10:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2104 T287230', diff saved to https://phabricator.wikimedia.org/P16925 and previous config saved to /var/cache/conftool/dbconfig/20210729-102753-marostegui.json |
[production] |
| 09:59 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@6960d32]: Increase mirrored traffic to tegola (duration: 00m 22s) |
[production] |
| 09:59 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@6960d32]: Increase mirrored traffic to tegola |
[production] |
| 09:47 |
<moritzm> |
installing Mariadb 10.3.29 updates from Buster point release (as packaged in Debian, not the WMF DB packages) |
[production] |
| 09:40 |
<jelto> |
uncordon kubestage1002.eqiad.wmnet as rsyslog was restarted and log shipping to logstash works again |
[production] |
| 09:14 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet |
[production] |
| 09:04 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet |
[production] |
| 08:54 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 08:51 |
<jmm@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 08:33 |
<moritzm> |
purging obsolete kernels from moscovium (disk space alerts for /) |
[production] |
| 08:13 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moscovium.eqiad.wmnet |
[production] |
| 08:09 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host moscovium.eqiad.wmnet |
[production] |
| 07:55 |
<elukey> |
roll restart uwsgi + celery on ores[12]* nodes to pick up aspell upgrades |
[production] |
| 07:53 |
<mbsantos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 07:52 |
<moritzm> |
restarting Tomcat on idp-test |
[production] |
| 06:41 |
<XioNoX> |
push pfw policies - T287203 |
[production] |
| 05:44 |
<Amir1> |
adding "comunicaciones AT wikimediacolombia.org" as owner of wikimedia-co mailing list |
[production] |
| 01:08 |
<eileen> |
civicrm revision changed from 739c936298 to 158ed65e00, config revision is 6011d9c471 |
[production] |
|
2021-07-28
§
|
| 23:57 |
<thcipriani@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:708581|wgSkipSkins: Update defaults, hide modern (T287616)]] (duration: 01m 06s) |
[production] |
| 23:50 |
<thcipriani@deploy1002> |
Synchronized wmf-config: Config: [[gerrit:708158|Disable mobile contributions simplifications on Wikidata and Commons (T283988)]] (duration: 01m 58s) |
[production] |
| 19:16 |
<twentyafterfour@deploy1002> |
Synchronized php: group1 wikis to 1.37.0-wmf.16 refs T281157 (duration: 01m 06s) |
[production] |
| 19:15 |
<twentyafterfour@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.16 refs T281157 |
[production] |
| 19:09 |
<twentyafterfour> |
Preparing to deploy 1.37.0-wmf.16 to group1 wikis |
[production] |
| 18:57 |
<legoktm> |
mwmaint2002$ foreachwikiindblist wikimania refreshLinks.php - to start populating DPL tracking category |
[production] |
| 18:36 |
<legoktm@deploy1002> |
Finished scap: Add a tracking category to pages using the <DynamicPageList> tag (duration: 27m 16s) |
[production] |
| 18:14 |
<jbond> |
manually cleared out the puppetdb2002 queue |
[production] |
| 18:08 |
<legoktm@deploy1002> |
Started scap: Add a tracking category to pages using the <DynamicPageList> tag |
[production] |
| 16:37 |
<ryankemper> |
[WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good |
[production] |
| 16:00 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 15:59 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 15:59 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 15:58 |
<ryankemper> |
T287112 [WDQS] Re-pooled `wdqs2002` |
[production] |
| 15:57 |
<ryankemper@deploy1002> |
Finished deploy [wdqs/wdqs@26273d8]: 0.3.77 (duration: 08m 55s) |
[production] |
| 15:53 |
<mutante> |
mw1434,mw1435,mw1436: scap pull, repooled, reimaged, converted from API to appserver for balancing (T279309) |
[production] |
| 15:53 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw143[4-6].eqiad.wmnet |
[production] |
| 15:52 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw143[4-6].eqiad.wmnet |
[production] |
| 15:51 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.77` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 15:48 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@26273d8]: 0.3.77 |
[production] |
| 15:47 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.77`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |
| 15:47 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 15:08 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:05 |
<jmm@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:58 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1434.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:56 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1434.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:39 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |