|
2021-01-05
ยง
|
| 18:21 |
<mbsantos@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:18 |
<mbsantos@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:13 |
<elukey> |
run homer on cr1/cr2-eqiad to update the analytics-in4 filter (https://gerrit.wikimedia.org/r/c/operations/homer/public/+/654469) |
[production] |
| 18:08 |
<mbsantos@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 17:10 |
<longma> |
1.36.0-wmf.25 was branched at 083fd09afcd204cfef177e11d7a5e4fd1217acfc for T267418 |
[production] |
| 17:00 |
<XioNoX> |
capture packets on pfw3-eqiad:reth0.1134 - T263833 |
[production] |
| 15:50 |
<jbond42> |
merging puppetlabs-lvm update |
[production] |
| 15:41 |
<volans> |
upgraded wmflib to 0.0.6 on all hosts where it's installed - T257905 |
[production] |
| 15:37 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2025.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:35 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc2025.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:35 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1025.eqiad.wmnet with reason: REIMAGE |
[production] |
| 15:33 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1025.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:59 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove overrides from wgEventLoggingSchemas (duration: 00m 57s) |
[production] |
| 13:40 |
<moritzm> |
installing python-apt security updates on buster/stretch |
[production] |
| 13:29 |
<moritzm> |
installing xen security updates on buster |
[production] |
| 13:01 |
<moritzm> |
installing lxml security updates for stretch |
[production] |
| 12:48 |
<elukey> |
add PXE d-i rescue bootable image config for jessie/stretch/buster to tftp |
[production] |
| 12:43 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:29 |
<jmm@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:13 |
<sukhe@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on malmok.wikimedia.org with reason: rebooting for kernel update |
[production] |
| 12:13 |
<sukhe@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:10:00 on malmok.wikimedia.org with reason: rebooting for kernel update |
[production] |
| 12:12 |
<moritzm> |
installing p11-kit security updates on buster |
[production] |
| 12:01 |
<marostegui> |
Restart db2121 T271106 |
[production] |
| 11:53 |
<moritzm> |
installing lxml security updates for buster |
[production] |
| 11:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 100%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13656 and previous config saved to /var/cache/conftool/dbconfig/20210105-110246-root.json |
[production] |
| 10:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 10:49 |
<jmm@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 75%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13655 and previous config saved to /var/cache/conftool/dbconfig/20210105-104742-root.json |
[production] |
| 10:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 50%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13654 and previous config saved to /var/cache/conftool/dbconfig/20210105-103239-root.json |
[production] |
| 10:26 |
<godog> |
swift codfw-prod: more weight to ms-be20[58-61] - T269337 |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 25%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13653 and previous config saved to /var/cache/conftool/dbconfig/20210105-101735-root.json |
[production] |
| 10:02 |
<hnowlan> |
stopping stray cpjobqueue processes on scb hosts |
[production] |
| 09:46 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:39 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:21 |
<ema> |
cp3054: upgrade varnish to 6.0.1-1wm1 T264398 |
[production] |
| 08:56 |
<moritzm> |
installing flac security updates |
[production] |
| 08:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2140 after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P13652 and previous config saved to /var/cache/conftool/dbconfig/20210105-084807-marostegui.json |
[production] |
| 08:32 |
<elukey> |
reboot sretest1001 to test some new PXE rescue settings |
[production] |
| 08:30 |
<marostegui> |
Restart db2127 T271106 |
[production] |
| 08:27 |
<hashar> |
Restarted CI Jenkins on contint2001 |
[production] |
| 07:14 |
<elukey> |
execute 'apt-get clean' on an-airflow1001 to recover disk space (root partition almost saturated) |
[production] |
| 06:41 |
<marostegui> |
Stop MySQL on db1074 - this will generate lag on s2 on labs |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1074 to clone db1155:3312 T268742 ', diff saved to https://phabricator.wikimedia.org/P13647 and previous config saved to /var/cache/conftool/dbconfig/20210105-064026-marostegui.json |
[production] |
| 03:42 |
<eileen> |
eoy receipts off to investigate issue ds has hit with Japanese names process-control config revision is d8756a45c1 |
[production] |
| 02:55 |
<legoktm@deploy1001> |
Synchronized php-1.36.0-wmf.22/extensions/AbuseFilter/: Rename maintenance/purgeOldLogIPData.php script (T271182) (duration: 00m 59s) |
[production] |
| 02:20 |
<ryankemper> |
[wdqs deploy] Deploy completed without issue |
[production] |
| 01:51 |
<ryankemper> |
[wdqs deploy] Restarting `wdqs-categories` across non-test wdqs nodes one at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 01:50 |
<ryankemper> |
[wdqs deploy] Restarted categories across all wdqs test instances: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 01:50 |
<ryankemper> |
[wdqs deploy] Restarted `wdqs-updater` across the whole fleet simultaneously: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 01:48 |
<ryankemper@deploy1001> |
Finished deploy [wdqs/wdqs@0432f8c]: 0.3.57 (duration: 08m 44s) |
[production] |