|
2021-01-05
ยง
|
| 18:49 |
<bstorm> |
changing the limits on k8s etcd nodes again, so disabling puppet on them T267966 |
[tools] |
| 18:22 |
<jhuneidi@deploy1001> |
Started scap: testwikis wikis to 1.36.0-wmf.25 refs T267418 |
[production] |
| 18:21 |
<mbsantos@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:18 |
<mbsantos@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:13 |
<elukey> |
run homer on cr1/cr2-eqiad to update the analytics-in4 filter (https://gerrit.wikimedia.org/r/c/operations/homer/public/+/654469) |
[production] |
| 18:08 |
<mbsantos@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 17:10 |
<longma> |
1.36.0-wmf.25 was branched at 083fd09afcd204cfef177e11d7a5e4fd1217acfc for T267418 |
[production] |
| 17:00 |
<XioNoX> |
capture packets on pfw3-eqiad:reth0.1134 - T263833 |
[production] |
| 15:50 |
<jbond42> |
merging puppetlabs-lvm update |
[production] |
| 15:41 |
<volans> |
upgraded wmflib to 0.0.6 on all hosts where it's installed - T257905 |
[production] |
| 15:37 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2025.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:35 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc2025.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:35 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1025.eqiad.wmnet with reason: REIMAGE |
[production] |
| 15:33 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1025.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:59 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove overrides from wgEventLoggingSchemas (duration: 00m 57s) |
[production] |
| 14:11 |
<dcaro> |
finished ssl tests for enc, cleaned up cloud-puppetmaster-03 (T268877) |
[cloudinfra] |
| 13:40 |
<moritzm> |
installing python-apt security updates on buster/stretch |
[production] |
| 13:29 |
<moritzm> |
installing xen security updates on buster |
[production] |
| 13:07 |
<dcaro> |
adding custom nginx config for labspuppetbackend on cloud-puppetmaster-03 to test ssl (T268877) |
[cloudinfra] |
| 13:01 |
<moritzm> |
installing lxml security updates for stretch |
[production] |
| 12:48 |
<elukey> |
add PXE d-i rescue bootable image config for jessie/stretch/buster to tftp |
[production] |
| 12:43 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:41 |
<arturo> |
live-hacking cloudinfra-internal-puppetmaster02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/654415 (T260834) |
[cloudinfra] |
| 12:31 |
<arturo> |
refresh acme-chief config for mx certs https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/949f1b4e81f3a1c6d4f4825292343f1ee17c48a1%5E%21/ (T260834) |
[cloudinfra] |
| 12:29 |
<jmm@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:21 |
<arturo> |
resolve git merge conflicts and rebase cloudinfra-internal-puppetmaster-02 /var/lib/git/labs/private |
[cloudinfra] |
| 12:13 |
<sukhe@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on malmok.wikimedia.org with reason: rebooting for kernel update |
[production] |
| 12:13 |
<sukhe@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:10:00 on malmok.wikimedia.org with reason: rebooting for kernel update |
[production] |
| 12:12 |
<moritzm> |
installing p11-kit security updates on buster |
[production] |
| 12:12 |
<arturo> |
created puppet prefix `mx-out` and added hiera to use internal puppetmaster (T260834) |
[cloudinfra] |
| 12:01 |
<marostegui> |
Restart db2121 T271106 |
[production] |
| 11:53 |
<moritzm> |
installing lxml security updates for buster |
[production] |
| 11:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 100%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13656 and previous config saved to /var/cache/conftool/dbconfig/20210105-110246-root.json |
[production] |
| 10:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 10:49 |
<jmm@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 75%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13655 and previous config saved to /var/cache/conftool/dbconfig/20210105-104742-root.json |
[production] |
| 10:40 |
<dcaro> |
removing dumps-[1..*] backups from cloudvirt1024 as they are not needed (T271094) |
[admin] |
| 10:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 50%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13654 and previous config saved to /var/cache/conftool/dbconfig/20210105-103239-root.json |
[production] |
| 10:26 |
<godog> |
swift codfw-prod: more weight to ms-be20[58-61] - T269337 |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 25%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13653 and previous config saved to /var/cache/conftool/dbconfig/20210105-101735-root.json |
[production] |
| 10:02 |
<hnowlan> |
stopping stray cpjobqueue processes on scb hosts |
[production] |
| 09:46 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:39 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:29 |
<joal> |
Manually reload unique-devices monthly in cassandra to fix T271170 |
[analytics] |
| 09:21 |
<ema> |
cp3054: upgrade varnish to 6.0.1-1wm1 T264398 |
[production] |
| 08:56 |
<moritzm> |
installing flac security updates |
[production] |
| 08:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2140 after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P13652 and previous config saved to /var/cache/conftool/dbconfig/20210105-084807-marostegui.json |
[production] |
| 08:32 |
<elukey> |
reboot sretest1001 to test some new PXE rescue settings |
[production] |
| 08:30 |
<marostegui> |
Restart db2127 T271106 |
[production] |
| 08:27 |
<hashar> |
Restarted CI Jenkins on contint2001 |
[production] |