1201-1250 of 10000 results (77ms)
2023-03-20 §
08:15 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 138915 [production]
08:00 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 138915 [production]
2023-03-19 §
18:27 <AndyRussG> update config (to re-enable old PayPal orphan slayer job) 27a5b481 -> 6359222d [production]
16:44 <apergos> dumpsdata1005 conversion to primary dumps nfs server done [production]
15:12 <AndyRussG> update config (to disable paypal_ec pending transaction resolver) 5dd37c9c -> 3d3606f1 [production]
14:18 <apergos> work starting now to swap dumpsdata1005 in for primary nfs server, replacing dumpsdata1003 which will become dumps spare host [production]
00:17 <fab@deploy2002> Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 05s) [production]
00:17 <fab@deploy2002> Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) [production]
2023-03-18 §
22:47 <fab@deploy2002> Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 19s) [production]
22:47 <fab@deploy2002> Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) [production]
14:26 <apergos> rsync of xmldata public dir from screen as ariel on dumpsdata1004 to dumpsdata1005, no bandwidth cap [production]
13:46 <apergos> rsync of xmldata private dir from screen as ariel on dumpsdata1004 to dumpsdata1005, no bandwidth cap [production]
07:55 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cephosd[1001-1005].eqiad.wmnet with reason: Systemd units failing, pupper tries to bring them up periodically, spam on IRC [production]
07:55 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on cephosd[1001-1005].eqiad.wmnet with reason: Systemd units failing, pupper tries to bring them up periodically, spam on IRC [production]
02:57 <fab@deploy2002> Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 05s) [production]
02:57 <fab@deploy2002> Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) [production]
01:20 <urandom> powercycling restbase2025 — T332462 [production]
00:06 <AndyRussG> Updating civicrm from 5dd37c9c to 3d3606f1 [production]
2023-03-17 §
19:53 <ebernhardson@deploy2002> Finished deploy [airflow-dags/search@4aeffc6]: improve handling of ores threshold fetching (duration: 00m 13s) [production]
19:53 <ebernhardson@deploy2002> Started deploy [airflow-dags/search@4aeffc6]: improve handling of ores threshold fetching [production]
19:52 <bd808> Testing Mastodon account changes. This should post to @wikimedia_sal@botsin.space [production]
19:06 <ebernhardson@deploy2002> Finished deploy [airflow-dags/search@7d75578]: enable templating of ores threshold fetch (duration: 00m 13s) [production]
19:06 <ebernhardson@deploy2002> Started deploy [airflow-dags/search@7d75578]: enable templating of ores threshold fetch [production]
18:35 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs6002.drmrs.wmnet with reason: rebooting for kernel updates [production]
18:35 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:30:00 on lvs6002.drmrs.wmnet with reason: rebooting for kernel updates [production]
18:34 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs5005.eqsin.wmnet with reason: rebooting for kernel updates [production]
18:34 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:30:00 on lvs5005.eqsin.wmnet with reason: rebooting for kernel updates [production]
18:32 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:40:00 on lvs1017.eqiad.wmnet with reason: rebooting for kernel updates [production]
18:31 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:40:00 on lvs1017.eqiad.wmnet with reason: rebooting for kernel updates [production]
18:10 <fab@deploy2002> Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 19s) [production]
18:09 <fab@deploy2002> Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) [production]
18:04 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs2007.codfw.wmnet with reason: rebooting for kernel updates [production]
18:04 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:30:00 on lvs2007.codfw.wmnet with reason: rebooting for kernel updates [production]
17:35 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs6001.drmrs.wmnet with reason: rebooting for kernel updates [production]
17:35 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:30:00 on lvs6001.drmrs.wmnet with reason: rebooting for kernel updates [production]
17:31 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5004.eqsin.wmnet [production]
17:31 <sukhe@cumin2002> START - Cookbook sre.hosts.remove-downtime for lvs5004.eqsin.wmnet [production]
17:29 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs4008.ulsfo.wmnet with reason: rebooting for kernel updates [production]
17:29 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:30:00 on lvs4008.ulsfo.wmnet with reason: rebooting for kernel updates [production]
17:05 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs5004.eqsin.wmnet with reason: rebooting for kernel updates [production]
17:05 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:30:00 on lvs5004.eqsin.wmnet with reason: rebooting for kernel updates [production]
15:50 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
15:29 <bking@cumin1001> START - Cookbook sre.wdqs.restart [production]
15:24 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
14:55 <bking@cumin1001> START - Cookbook sre.wdqs.restart [production]
14:55 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) [production]
14:55 <bking@cumin1001> START - Cookbook sre.wdqs.restart [production]
14:54 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) [production]
14:54 <bking@cumin1001> START - Cookbook sre.wdqs.restart [production]
14:35 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]