6701-6750 of 10000 results (115ms)
2023-03-20 §
13:34 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:30:00 on lvs3005.esams.wmnet with reason: rebooting for kernel updates [production]
13:30 <awight@deploy2002> Finished deploy [kartotherian/deploy@906be32] (eqiad): Update kartotherian to a6e9843 (duration: 01m 30s) [production]
13:29 <samtar@deploy2002> stang and samtar: Backport for [[gerrit:900689|kuwiktionary: Add wordmark (T326067)]], [[gerrit:900742|trwikivoyage: Update wordmark (T332439)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
13:29 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cuminunpriv1001.eqiad.wmnet with reason: host reimage [production]
13:29 <awight@deploy2002> Started deploy [kartotherian/deploy@906be32] (eqiad): Update kartotherian to a6e9843 [production]
13:28 <samtar@deploy2002> Started scap: Backport for [[gerrit:900689|kuwiktionary: Add wordmark (T326067)]], [[gerrit:900742|trwikivoyage: Update wordmark (T332439)]] [production]
13:28 <kharlan@deploy2002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
13:26 <awight@deploy2002> Finished deploy [kartotherian/deploy@906be32] (codfw): Update kartotherian to a6e9843 (duration: 01m 39s) [production]
13:26 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cuminunpriv1001.eqiad.wmnet with reason: host reimage [production]
13:24 <awight@deploy2002> Started deploy [kartotherian/deploy@906be32] (codfw): Update kartotherian to a6e9843 [production]
13:18 <samtar@deploy2002> Finished scap: Backport for [[gerrit:900537|bewiki: Remove group "autoeditor", "reviewer" (T326012)]], [[gerrit:900690|slwiki: Create Draft namespace (T332351)]] (duration: 11m 36s) [production]
13:18 <jmm@cumin2002> START - Cookbook sre.ganeti.reimage for host cuminunpriv1001.eqiad.wmnet with OS bullseye [production]
13:17 <kharlan@deploy2002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
13:17 <kharlan@deploy2002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
13:15 <kharlan@deploy2002> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [production]
13:14 <kharlan@deploy2002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
13:14 <kharlan@deploy2002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [production]
13:08 <samtar@deploy2002> stang and samtar: Backport for [[gerrit:900537|bewiki: Remove group "autoeditor", "reviewer" (T326012)]], [[gerrit:900690|slwiki: Create Draft namespace (T332351)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
13:06 <samtar@deploy2002> Started scap: Backport for [[gerrit:900537|bewiki: Remove group "autoeditor", "reviewer" (T326012)]], [[gerrit:900690|slwiki: Create Draft namespace (T332351)]] [production]
11:35 <krinkle@deploy2002> Synchronized php-1.40.0-wmf.27/includes/libs/rdbms/: (no justification provided) (duration: 15m 28s) [production]
09:57 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 36692 [production]
09:56 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 36692 [production]
09:56 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 12956 [production]
09:56 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 12956 [production]
09:55 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 141082 [production]
09:55 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 141082 [production]
09:55 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 58655 [production]
09:54 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 58655 [production]
09:54 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 2552 [production]
09:54 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 2552 [production]
09:21 <claime> Repooling parse2004 - T332119 [production]
08:18 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'show' for AS: 138915 [production]
08:18 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'show' for AS: 138915 [production]
08:15 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 138915 [production]
08:00 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 138915 [production]
2023-03-19 §
18:27 <AndyRussG> update config (to re-enable old PayPal orphan slayer job) 27a5b481 -> 6359222d [production]
16:44 <apergos> dumpsdata1005 conversion to primary dumps nfs server done [production]
15:12 <AndyRussG> update config (to disable paypal_ec pending transaction resolver) 5dd37c9c -> 3d3606f1 [production]
14:18 <apergos> work starting now to swap dumpsdata1005 in for primary nfs server, replacing dumpsdata1003 which will become dumps spare host [production]
00:17 <fab@deploy2002> Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 05s) [production]
00:17 <fab@deploy2002> Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) [production]
2023-03-18 §
22:47 <fab@deploy2002> Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 19s) [production]
22:47 <fab@deploy2002> Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) [production]
14:26 <apergos> rsync of xmldata public dir from screen as ariel on dumpsdata1004 to dumpsdata1005, no bandwidth cap [production]
13:46 <apergos> rsync of xmldata private dir from screen as ariel on dumpsdata1004 to dumpsdata1005, no bandwidth cap [production]
07:55 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cephosd[1001-1005].eqiad.wmnet with reason: Systemd units failing, pupper tries to bring them up periodically, spam on IRC [production]
07:55 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on cephosd[1001-1005].eqiad.wmnet with reason: Systemd units failing, pupper tries to bring them up periodically, spam on IRC [production]
02:57 <fab@deploy2002> Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 05s) [production]
02:57 <fab@deploy2002> Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) [production]
01:20 <urandom> powercycling restbase2025 — T332462 [production]