251-300 of 10000 results (107ms)
2024-12-23 §
07:46 <akosiaris@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1290.eqiad.wmnet [production]
07:03 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on dbproxy1029.eqiad.wmnet with reason: maintenance [production]
07:03 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 10 days, 0:00:00 on dbproxy1029.eqiad.wmnet with reason: maintenance [production]
07:03 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on dbproxy1028.eqiad.wmnet with reason: maintenance [production]
07:03 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 10 days, 0:00:00 on dbproxy1028.eqiad.wmnet with reason: maintenance [production]
2024-12-22 §
11:43 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [production]
11:40 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [production]
10:52 <Emperor> restart swift-object on ms-be2082 [production]
10:49 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [production]
10:46 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [production]
2024-12-21 §
00:00 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
00:00 <andrew@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002" [production]
2024-12-20 §
22:12 <andrew@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002" [production]
21:52 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd2004-dev.codfw.wmnet with reason: host reimage [production]
21:48 <andrew@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd2004-dev.codfw.wmnet with reason: host reimage [production]
21:29 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
21:24 <andrew@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
21:13 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
18:43 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71741 and previous config saved to /var/cache/conftool/dbconfig/20241220-184307-root.json [production]
18:35 <kamila@cumin1002> END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1023-1024].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]
18:35 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1024.eqiad.wmnet with OS bookworm [production]
18:28 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71739 and previous config saved to /var/cache/conftool/dbconfig/20241220-182801-root.json [production]
18:15 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1024.eqiad.wmnet with reason: host reimage [production]
18:12 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71738 and previous config saved to /var/cache/conftool/dbconfig/20241220-181256-root.json [production]
18:12 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1024.eqiad.wmnet with reason: host reimage [production]
18:02 <xcollazo@deploy2002> Finished deploy [airflow-dags/analytics@7fecc64]: Pickup hotfix for T377852. (duration: 02m 03s) [production]
18:00 <xcollazo@deploy2002> Started deploy [airflow-dags/analytics@7fecc64]: Pickup hotfix for T377852. [production]
17:57 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71737 and previous config saved to /var/cache/conftool/dbconfig/20241220-175751-root.json [production]
17:53 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1024.eqiad.wmnet with OS bookworm [production]
17:52 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1023.eqiad.wmnet with OS bookworm [production]
17:42 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71736 and previous config saved to /var/cache/conftool/dbconfig/20241220-174246-root.json [production]
17:39 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2168', diff saved to https://phabricator.wikimedia.org/P71735 and previous config saved to /var/cache/conftool/dbconfig/20241220-173922-marostegui.json [production]
17:38 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db2168.codfw.wmnet with reason: maintenance [production]
17:38 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on db2168.codfw.wmnet with reason: maintenance [production]
17:31 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1023.eqiad.wmnet with reason: host reimage [production]
17:26 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1023.eqiad.wmnet with reason: host reimage [production]
17:08 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1023.eqiad.wmnet with OS bookworm [production]
17:06 <kamila@cumin1002> START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1023-1024].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]
16:48 <xcollazo@deploy2002> Finished deploy [airflow-dags/analytics@8c5744d]: Deploying latest analytics Airflow instance DAGs. T377852. (duration: 00m 58s) [production]
16:47 <xcollazo@deploy2002> Started deploy [airflow-dags/analytics@8c5744d]: Deploying latest analytics Airflow instance DAGs. T377852. [production]
14:38 <btullis@cumin1002> END (FAIL) - Cookbook sre.hadoop.roll-restart-masters (exit_code=99) restart masters for Hadoop analytics cluster: Restart of jvm daemons. [production]
14:06 <btullis@cumin1002> START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop analytics cluster: Restart of jvm daemons. [production]
12:10 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1001-1004].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]
12:10 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1004.eqiad.wmnet with OS bookworm [production]
12:08 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1008-1011].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]
12:08 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1011.eqiad.wmnet with OS bookworm [production]
11:52 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1004.eqiad.wmnet with reason: host reimage [production]
11:48 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1011.eqiad.wmnet with reason: host reimage [production]
11:47 <jayme@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1004.eqiad.wmnet with reason: host reimage [production]
11:45 <jayme@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1011.eqiad.wmnet with reason: host reimage [production]