501-550 of 10000 results (18ms)
2020-12-10 §
11:56 <dcaro> Freed some space on cloudvirt1024 by running the purge script (T269419) [admin]
11:50 <arturo> restarted tool, the bot left all IRC channels... [tools.stashbot]
11:50 <arturo> disabled puppet in all k8s nodes in preparation for version upgrade (T263284) [tools]
11:44 <arturo> disabled puppet in all k8s nodes in preparation for version upgrade (T263284) [tools]
11:21 <effie> upload rometheus-redis-exporter_0.13-1 to buster-wikimedia main [production]
11:20 <moritzm> installing apt security updates on buster/stretch [production]
11:13 <elukey@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:07 <elukey@cumin1001> START - Cookbook sre.dns.netbox [production]
11:06 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
11:05 <moritzm> rebooting failoid1001 [production]
11:04 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
10:45 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
10:43 <godog> swift eqiad-prod: add weight to ms-be106[0-3] - T268435 [production]
10:42 <jbond@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
10:38 <effie> uploading prometheus-redis-exporter_0.13-1 in component/redis2 for buster [production]
10:37 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
10:37 <jbond@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) [production]
10:33 <jbond@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
10:33 <jbond@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) [production]
10:33 <jbond@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
10:32 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
10:29 <volans> upgraded spicearack to 0.0.46 on cumin[12]001 [production]
10:16 <volans> uploaded spicerack_0.0.46 to apt.wikimedia.org buster-wikimedia [production]
09:58 <dcaro> successful tesseract upgrade on tools-sgewebgrid-lighttpd-0914, upgrading the rest of nodes (clinic duty) [tools]
09:57 <ema> A:cp rolling ats-{tls,backend}-restart for openssl upgrades (CVE-2020-1971) [production]
09:49 <dcaro> upgrading tesseract on tools-sgewebgrid-lighttpd-0914 (clinic duty) [tools]
09:28 <ema> cp3054: downgrade varnish to 6.0.0-1wm1 T264398 [production]
09:28 <effie> disable puppet on all hosts running nutcracker for 647204 - T265643 [production]
09:26 <effie> disable puppet on all mw* hosts for 647204 - T265643 [production]
09:17 <dcaro> removing leaked dns record discordwiki.eqiad.wmflabs (clinic duty) [admin]
09:17 <dcaro> removing leaked dns record discordwiki.srwiki-dev.eqiad.wmflabs (clinic duty) [srwiki-dev]
09:11 <effie> disable puppet on all hosts running redis - T265643 [production]
08:58 <dcaro> starting a new etcd instance completely from ansible playbook (etcd-8) (T267412) [toolsbeta]
08:22 <godog> swift codfw-prod: more weight to ms-be20[58-61] - T269337 [production]
06:38 <kart_> Upgraded Apertium to 2020-12-09-115733-production [production]
06:35 <kartik@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'production' . [production]
06:35 <kartik@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'plain' . [production]
06:30 <kartik@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'plain' . [production]
06:30 <kartik@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'production' . [production]
06:27 <kartik@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'staging' . [production]
00:42 <ejegg> updated payments-wiki from 756c2f7ce0 to df80a99b40 [production]
00:26 <robh> cr2-eqsin bad fan being swapped via T267544 [production]
00:05 <mutante> re-enabling puppet (disabled without reason) on wikistats-wild-tiger [wikistats]
2020-12-09 §
23:21 <mutante> repooling parse2001 after buster reimage - T268524 [production]
23:19 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2243.codfw.wmnet [production]
23:16 <mutante> repooling parse2001 after buster reimage - T245757 [production]
23:15 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=parse2001.codfw.wmnet [production]
23:04 <mutante> zero.wikimedia.beta.wmflabs.org removed from beta_sites (deployment-prep) T187716 [production]
22:32 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
22:30 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]