1851-1900 of 10000 results (39ms)
2021-11-17 §
09:39 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6016.drmrs.wmnet with OS buster [production]
09:35 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6014.drmrs.wmnet with OS buster [production]
09:19 <_joe_> removing php 7.3 images from docker-registry.wikimedia.org [production]
09:13 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6015.drmrs.wmnet with OS buster [production]
09:11 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6013.drmrs.wmnet with OS buster [production]
09:03 <moritzm> installing ffmpeg security updates on stretch [production]
09:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 100%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17752 and previous config saved to /var/cache/conftool/dbconfig/20211117-090124-root.json [production]
08:56 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6014.drmrs.wmnet with OS buster [production]
08:54 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6012.drmrs.wmnet with OS buster [production]
08:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 75%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17751 and previous config saved to /var/cache/conftool/dbconfig/20211117-084621-root.json [production]
08:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 50%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17750 and previous config saved to /var/cache/conftool/dbconfig/20211117-083117-root.json [production]
08:30 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6013.drmrs.wmnet with OS buster [production]
08:24 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6011.drmrs.wmnet with OS buster [production]
08:16 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 40%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17749 and previous config saved to /var/cache/conftool/dbconfig/20211117-081613-root.json [production]
08:14 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6012.drmrs.wmnet with OS buster [production]
08:11 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6010.drmrs.wmnet with OS buster [production]
08:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 25%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17748 and previous config saved to /var/cache/conftool/dbconfig/20211117-080110-root.json [production]
07:49 <elukey> restart coal, navtiming, statsv (refreshed by puppet) after https://gerrit.wikimedia.org/r/737970 [production]
07:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 20%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17747 and previous config saved to /var/cache/conftool/dbconfig/20211117-074606-root.json [production]
07:44 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6011.drmrs.wmnet with OS buster [production]
07:34 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6009.drmrs.wmnet with OS buster [production]
07:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 10%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17746 and previous config saved to /var/cache/conftool/dbconfig/20211117-073102-root.json [production]
07:31 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6010.drmrs.wmnet with OS buster [production]
07:29 <elukey> `apt-get clean` on an-tool1005 to free space in the root partition [production]
07:28 <elukey> `sudo pkill -U jmixter` on stat100[5,8] to allow puppet to run and remove the offboarded user [production]
07:22 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6008.drmrs.wmnet with OS buster [production]
07:20 <Amir1> start of clean up of autreview logs of ruwiki, deleting 3.5M rows (T285608) [production]
07:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 5%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17745 and previous config saved to /var/cache/conftool/dbconfig/20211117-071559-root.json [production]
07:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 1%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P17744 and previous config saved to /var/cache/conftool/dbconfig/20211117-070055-root.json [production]
06:58 <marostegui> Upgrade db1180 to 10.4.22 [production]
06:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1180 for upgrade', diff saved to https://phabricator.wikimedia.org/P17743 and previous config saved to /var/cache/conftool/dbconfig/20211117-065740-marostegui.json [production]
06:52 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6009.drmrs.wmnet with OS buster [production]
06:43 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6008.drmrs.wmnet with OS buster [production]
06:38 <Amir1> start of deleting auto-review logs in arwiki (T285608) deleting 23M rows [production]
06:33 <marostegui> Upgrade clouddb1018 [production]
06:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove logpager from s5 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17742 and previous config saved to /var/cache/conftool/dbconfig/20211117-060426-marostegui.json [production]
03:16 <eileen> checkout revision (c67b18b9 -> 9c5f0b69) [production]
02:10 <eileen> * revision 817e514a -> c67b18b9 (latest) civicrm [production]
00:19 <ryankemper> T276198 `ryankemper@cumin1001:~$ sudo cumin -b 3 '*elastic*' 'sudo run-puppet-agent --force'` Change looks good (no complaints from systemd), rolling out to rest of fleet / reenabling puppet [production]
00:15 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor1006.eqiad.wmnet [production]
00:06 <legoktm@cumin1001> START - Cookbook sre.hosts.reboot-single for host thumbor1006.eqiad.wmnet [production]
2021-11-16 §
23:59 <ryankemper> T276198 `ryankemper@elastic1049:~$ sudo run-puppet-agent --force` to test out https://gerrit.wikimedia.org/r/c/operations/puppet/+/739375 [production]
23:57 <ejegg> updated payments-wiki from 49ad5962 -> abb2bd9d [production]
23:27 <ryankemper> T276198 `ryankemper@elastic1049:~$ sudo run-puppet-agent --force`; `elasticsearch_6@production-search-eqiad.service ` didn't restart but it looks like there might be slightly wrong with the new `ExecPreStart` line => `Executable path is not absolute, ignoring: systemd-tmpfiles --create /usr/lib/tmpfiles.d/elasticsearch.conf` [production]
23:27 <legoktm@cumin1001> conftool action : set/weight=5; selector: name=thumbor1005.eqiad.wmnet [production]
23:25 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=thumbor1005.eqiad.wmnet [production]
23:22 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=thumbor1005.eqiad.wmnet [production]
23:22 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=thumbor1005.eqiad.wmnet [production]
23:21 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=thumbor1005.eqiad.wmnet [production]
23:19 <ryankemper> T276198 `ryankemper@cumin1001:~$ sudo cumin '*elastic*' 'sudo disable-puppet "Merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/721644"'` (done a few mins ago) [production]