951-1000 of 10000 results (30ms)
2024-02-14 §
08:22 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71 [tools]
08:22 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71 [tools]
08:21 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [tools]
08:21 <taavi@cloudcumin1001> Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster [tools]
08:20 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1001.wikimedia.org [production]
08:16 <taavi> reboot clouddumps1001 for kernel updates [admin]
08:12 <taavi> restart apache2 on lists1001 to remove traces of old, soon-to-expire TLS certificate [production]
08:11 <taavi@cumin1002> START - Cookbook sre.hosts.reboot-single for host clouddumps1001.wikimedia.org [production]
08:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P56741 and previous config saved to /var/cache/conftool/dbconfig/20240214-081051-ladsgroup.json [production]
08:09 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster [tools]
08:08 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70 [tools]
08:07 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70 [tools]
08:05 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [tools]
08:05 <taavi@cloudcumin1001> Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster [tools]
07:56 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster [tools]
07:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1199 (T352010)', diff saved to https://phabricator.wikimedia.org/P56740 and previous config saved to /var/cache/conftool/dbconfig/20240214-075545-ladsgroup.json [production]
07:54 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69 [tools]
07:54 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69 [tools]
07:53 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [tools]
07:53 <taavi@cloudcumin1001> Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster [tools]
07:51 <stran@deploy2002> helmfile [codfw] DONE helmfile.d/services/ipoid: apply [production]
07:50 <stran@deploy2002> helmfile [codfw] START helmfile.d/services/ipoid: apply [production]
07:50 <stran@deploy2002> helmfile [eqiad] DONE helmfile.d/services/ipoid: apply [production]
07:49 <stran@deploy2002> helmfile [eqiad] START helmfile.d/services/ipoid: apply [production]
07:48 <stran@deploy2002> helmfile [staging] DONE helmfile.d/services/ipoid: apply [production]
07:48 <stran@deploy2002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
07:48 <stran@deploy2002> helmfile [staging] DONE helmfile.d/services/ipoid: apply [production]
07:47 <stran@deploy2002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
07:44 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster [tools]
07:43 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68 [tools]
07:42 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68 [tools]
06:22 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase1035.eqiad.wmnet with OS bullseye [production]
06:22 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
03:11 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1199 (T352010)', diff saved to https://phabricator.wikimedia.org/P56739 and previous config saved to /var/cache/conftool/dbconfig/20240214-031125-ladsgroup.json [production]
03:11 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance [production]
03:11 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance [production]
03:11 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T352010)', diff saved to https://phabricator.wikimedia.org/P56738 and previous config saved to /var/cache/conftool/dbconfig/20240214-031103-ladsgroup.json [production]
02:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P56737 and previous config saved to /var/cache/conftool/dbconfig/20240214-025557-ladsgroup.json [production]
02:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P56736 and previous config saved to /var/cache/conftool/dbconfig/20240214-024050-ladsgroup.json [production]
02:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T352010)', diff saved to https://phabricator.wikimedia.org/P56735 and previous config saved to /var/cache/conftool/dbconfig/20240214-022544-ladsgroup.json [production]
01:44 <eileen> civicrm upgraded from 497e0899 to 3ee91f59 [production]
00:04 <dzahn@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncmonitor1001.eqiad.wmnet with reason: host reimage [production]
00:01 <dzahn@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on ncmonitor1001.eqiad.wmnet with reason: host reimage [production]
2024-02-13 §
23:55 <dzahn@cumin1002> START - Cookbook sre.hosts.reimage for host ncmonitor1001.eqiad.wmnet with OS bookworm [production]
21:50 <brennen> add kostajh to acl*phabricator per T355405 [releng]
21:23 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1190 (T352010)', diff saved to https://phabricator.wikimedia.org/P56734 and previous config saved to /var/cache/conftool/dbconfig/20240213-212343-ladsgroup.json [production]
21:23 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance [production]
21:23 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance [production]
21:23 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T352010)', diff saved to https://phabricator.wikimedia.org/P56733 and previous config saved to /var/cache/conftool/dbconfig/20240213-212321-ladsgroup.json [production]
21:08 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P56732 and previous config saved to /var/cache/conftool/dbconfig/20240213-210814-ladsgroup.json [production]