|
2021-01-05
§
|
| 15:35 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc2025.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:35 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1025.eqiad.wmnet with reason: REIMAGE |
[production] |
| 15:33 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1025.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:59 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove overrides from wgEventLoggingSchemas (duration: 00m 57s) |
[production] |
| 13:40 |
<moritzm> |
installing python-apt security updates on buster/stretch |
[production] |
| 13:29 |
<moritzm> |
installing xen security updates on buster |
[production] |
| 13:01 |
<moritzm> |
installing lxml security updates for stretch |
[production] |
| 12:48 |
<elukey> |
add PXE d-i rescue bootable image config for jessie/stretch/buster to tftp |
[production] |
| 12:43 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:29 |
<jmm@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:13 |
<sukhe@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on malmok.wikimedia.org with reason: rebooting for kernel update |
[production] |
| 12:13 |
<sukhe@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:10:00 on malmok.wikimedia.org with reason: rebooting for kernel update |
[production] |
| 12:12 |
<moritzm> |
installing p11-kit security updates on buster |
[production] |
| 12:01 |
<marostegui> |
Restart db2121 T271106 |
[production] |
| 11:53 |
<moritzm> |
installing lxml security updates for buster |
[production] |
| 11:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 100%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13656 and previous config saved to /var/cache/conftool/dbconfig/20210105-110246-root.json |
[production] |
| 10:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 10:49 |
<jmm@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 75%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13655 and previous config saved to /var/cache/conftool/dbconfig/20210105-104742-root.json |
[production] |
| 10:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 50%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13654 and previous config saved to /var/cache/conftool/dbconfig/20210105-103239-root.json |
[production] |
| 10:26 |
<godog> |
swift codfw-prod: more weight to ms-be20[58-61] - T269337 |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 25%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13653 and previous config saved to /var/cache/conftool/dbconfig/20210105-101735-root.json |
[production] |
| 10:02 |
<hnowlan> |
stopping stray cpjobqueue processes on scb hosts |
[production] |
| 09:46 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:39 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:21 |
<ema> |
cp3054: upgrade varnish to 6.0.1-1wm1 T264398 |
[production] |
| 08:56 |
<moritzm> |
installing flac security updates |
[production] |
| 08:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2140 after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P13652 and previous config saved to /var/cache/conftool/dbconfig/20210105-084807-marostegui.json |
[production] |
| 08:32 |
<elukey> |
reboot sretest1001 to test some new PXE rescue settings |
[production] |
| 08:30 |
<marostegui> |
Restart db2127 T271106 |
[production] |
| 08:27 |
<hashar> |
Restarted CI Jenkins on contint2001 |
[production] |
| 07:14 |
<elukey> |
execute 'apt-get clean' on an-airflow1001 to recover disk space (root partition almost saturated) |
[production] |
| 06:41 |
<marostegui> |
Stop MySQL on db1074 - this will generate lag on s2 on labs |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1074 to clone db1155:3312 T268742 ', diff saved to https://phabricator.wikimedia.org/P13647 and previous config saved to /var/cache/conftool/dbconfig/20210105-064026-marostegui.json |
[production] |
| 03:42 |
<eileen> |
eoy receipts off to investigate issue ds has hit with Japanese names process-control config revision is d8756a45c1 |
[production] |
| 02:55 |
<legoktm@deploy1001> |
Synchronized php-1.36.0-wmf.22/extensions/AbuseFilter/: Rename maintenance/purgeOldLogIPData.php script (T271182) (duration: 00m 59s) |
[production] |
| 02:20 |
<ryankemper> |
[wdqs deploy] Deploy completed without issue |
[production] |
| 01:51 |
<ryankemper> |
[wdqs deploy] Restarting `wdqs-categories` across non-test wdqs nodes one at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 01:50 |
<ryankemper> |
[wdqs deploy] Restarted categories across all wdqs test instances: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 01:50 |
<ryankemper> |
[wdqs deploy] Restarted `wdqs-updater` across the whole fleet simultaneously: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 01:48 |
<ryankemper@deploy1001> |
Finished deploy [wdqs/wdqs@0432f8c]: 0.3.57 (duration: 08m 44s) |
[production] |
| 01:41 |
<ryankemper> |
[wdqs deploy] Canary `wdqs1003` passing all tests following deploy, proceeding to rest of fleet |
[production] |
| 01:40 |
<ryankemper@deploy1001> |
Started deploy [wdqs/wdqs@0432f8c]: 0.3.57 |
[production] |
| 01:38 |
<ryankemper> |
[wdqs deploy] Pre-deploy tests are all passing, proceeding with deploy shortly |
[production] |
| 01:20 |
<jgleeson> |
updated process-control config revision to 276a8ff5b6 |
[production] |
| 00:40 |
<jgleeson> |
updated civicrm revision changed from bb8baac617 to 6be8a130df |
[production] |