1301-1350 of 10000 results (51ms)
2022-03-08 §
02:29 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:28 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:28 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
02:07 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:06 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:57 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@1c598f5]: (no justification provided) (duration: 00m 04s) [production]
01:57 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@1c598f5]: (no justification provided) [production]
01:32 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@1c598f5]: (no justification provided) (duration: 00m 08s) [production]
01:31 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@1c598f5]: (no justification provided) [production]
01:22 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@21af07c]: (no justification provided) (duration: 00m 07s) [production]
01:22 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@21af07c]: (no justification provided) [production]
01:11 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@c47e886]: (no justification provided) (duration: 00m 04s) [production]
01:11 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@c47e886]: (no justification provided) [production]
01:07 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@c47e886]: (no justification provided) (duration: 00m 08s) [production]
01:07 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@c47e886]: (no justification provided) [production]
00:34 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@c8a753b]: (no justification provided) (duration: 00m 07s) [production]
00:34 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@c8a753b]: (no justification provided) [production]
00:08 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@b5f7840]: (no justification provided) (duration: 00m 08s) [production]
00:08 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@b5f7840]: (no justification provided) [production]
2022-03-07 §
23:50 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on mx2001.wikimedia.org with reason: reboot [production]
23:50 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on mx2001.wikimedia.org with reason: reboot [production]
23:49 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on mx1001.wikimedia.org with reason: reboot [production]
23:49 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on mx1001.wikimedia.org with reason: reboot [production]
23:40 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on mirror1001.wikimedia.org with reason: reboot [production]
23:40 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on mirror1001.wikimedia.org with reason: reboot [production]
22:37 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudservices1003.wikimedia.org with OS bullseye [production]
22:28 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudservices1003.wikimedia.org with reason: host reimage [production]
22:26 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices1003.wikimedia.org with reason: host reimage [production]
22:25 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudservices1003.wikimedia.org with OS bullseye [production]
22:21 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices1003.wikimedia.org with OS bullseye [production]
22:20 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudservices1003.wikimedia.org with reason: host reimage [production]
22:18 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices1003.wikimedia.org with reason: host reimage [production]
21:49 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudservices1003.wikimedia.org with OS bullseye [production]
21:38 <urbanecm> UTC late B&C window done [production]
21:37 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:37 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.24/skins/Vector/includes/SkinVector.php: eac551c: Fix language alert regression (T302018) (duration: 00m 50s) [production]
21:36 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:22 <eileen> config aa7dcd88 -> 16fa8e1c [production]
20:39 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices1003.wikimedia.org with OS bullseye [production]
20:16 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudservices1003.wikimedia.org with reason: host reimage [production]
20:13 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices1003.wikimedia.org with reason: host reimage [production]
19:49 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudservices1003.wikimedia.org with OS bullseye [production]
18:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T300381)', diff saved to https://phabricator.wikimedia.org/P22016 and previous config saved to /var/cache/conftool/dbconfig/20220307-181310-marostegui.json [production]
17:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P22015 and previous config saved to /var/cache/conftool/dbconfig/20220307-175805-marostegui.json [production]
17:55 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage2002.codfw.wmnet [production]