9651-9700 of 10000 results (73ms)
2024-08-08 §
06:35 <jelto@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version [production]
05:13 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node [admin]
05:13 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) [admin]
02:19 <cstone> civicrm upgraded from d1f1d7bd to 686c7c5f [production]
00:48 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node [admin]
00:48 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) [admin]
00:48 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) [admin]
00:30 <rzl@deploy1003> Finished scap: https://gerrit.wikimedia.org/r/1060184 (duration: 02m 33s) [production]
00:29 <rzl@deploy1003> Started scap sync-world: https://gerrit.wikimedia.org/r/1060184 [production]
2024-08-07 §
21:23 <cstone> payments-wiki upgraded from 88500664 to a7f3301a [production]
21:19 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host vrts1003.eqiad.wmnet with OS bookworm [production]
21:03 <cstone> payments-wiki upgraded from 49a9e765 to 88500664 [production]
21:02 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on vrts1003.eqiad.wmnet with reason: host reimage [production]
20:57 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on vrts1003.eqiad.wmnet with reason: host reimage [production]
20:56 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gerrit1004.wikimedia.org with OS bookworm [production]
20:53 <milimetric@deploy1003> Finished deploy [airflow-dags/analytics@4cf9922]: (no justification provided) (duration: 00m 38s) [production]
20:53 <milimetric@deploy1003> Started deploy [airflow-dags/analytics@4cf9922]: (no justification provided) [production]
20:39 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gerrit1004.wikimedia.org with reason: host reimage [production]
20:39 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host vrts1003.eqiad.wmnet with OS bookworm [production]
20:37 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host vrts1003.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:35 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on gerrit1004.wikimedia.org with reason: host reimage [production]
20:21 <cjming> end of UTC late backport window [production]
20:17 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host gerrit1004.wikimedia.org with OS bookworm [production]
20:15 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gerrit1004.wikimedia.org with OS bookworm [production]
20:11 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node [admin]
20:11 <milimetric@deploy1003> Finished deploy [airflow-dags/analytics@049c09e]: (no justification provided) (duration: 00m 03s) [production]
20:11 <milimetric@deploy1003> Started deploy [airflow-dags/analytics@049c09e]: (no justification provided) [production]
20:11 <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=97) [admin]
20:10 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [admin]
20:08 <jclark@cumin1002> START - Cookbook sre.hosts.provision for host vrts1003.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:07 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node [admin]
20:06 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [admin]
20:04 <jclark@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:04 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt vrts1003 - jclark@cumin1002" [production]
20:04 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt vrts1003 - jclark@cumin1002" [production]
20:01 <jclark@cumin1002> START - Cookbook sre.dns.netbox [production]
19:59 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gerrit1004.wikimedia.org with reason: host reimage [production]
19:56 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add [admin]
19:55 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on gerrit1004.wikimedia.org with reason: host reimage [production]
19:53 <milimetric@deploy1003> Finished deploy [airflow-dags/analytics@049c09e]: (no justification provided) (duration: 00m 59s) [production]
19:52 <milimetric@deploy1003> Started deploy [airflow-dags/analytics@049c09e]: (no justification provided) [production]
19:52 <milimetric@deploy1003> Finished deploy [airflow-dags/analytics@216348d]: (no justification provided) (duration: 00m 47s) [production]
19:51 <milimetric@deploy1003> Started deploy [airflow-dags/analytics@216348d]: (no justification provided) [production]
19:47 <milimetric@deploy1003> Finished deploy [airflow-dags/analytics@049c09e]: Deploying new Browser General job (duration: 00m 02s) [production]
19:47 <milimetric@deploy1003> Started deploy [airflow-dags/analytics@049c09e]: Deploying new Browser General job [production]
19:46 <milimetric@deploy1003> Finished deploy [airflow-dags/analytics@049c09e]: Deploying new Browser General job (duration: 00m 41s) [production]
19:45 <milimetric> deploying airflow-dags/analytics for browser general daily dag [analytics]
19:45 <milimetric@deploy1003> Started deploy [airflow-dags/analytics@049c09e]: Deploying new Browser General job [production]
19:39 <ebernhardson@deploy1003> Finished deploy [airflow-dags/search@049c09e]: workaround process_sparql_query oom issues (duration: 00m 20s) [production]
19:39 <ebernhardson@deploy1003> Started deploy [airflow-dags/search@049c09e]: workaround process_sparql_query oom issues [production]