1-50 of 10000 results (22ms)
2024-12-22 §
11:43 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [production]
11:40 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [production]
10:52 <Emperor> restart swift-object on ms-be2082 [production]
10:49 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [production]
10:46 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [production]
2024-12-21 §
22:52 <wmbot~lucaswerkmeister@tools-bastion-13> deployed eb9d0ae3c2 (l10n updates: lb, pa; also upgrade dependencies, including Flask 3.1.0 and Jinja2 3.1.5) [tools.lexeme-forms]
00:00 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
00:00 <andrew@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002" [production]
2024-12-20 §
23:01 <brennen> brennen@releases1003:/srv/org/wikimedia/releases/mediawiki/1.43$ sudo cp -i ~atieno/mediawiki-* ./ [releng]
22:12 <andrew@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002" [production]
21:52 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd2004-dev.codfw.wmnet with reason: host reimage [production]
21:48 <andrew@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd2004-dev.codfw.wmnet with reason: host reimage [production]
21:29 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
21:24 <andrew@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
21:13 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
19:46 <btullis> restarted the hadoop-hdfs-namenode service on an-master1004 to pick up the new settins as well. [analytics]
19:46 <btullis> failed over the hadoop namenode services from an-master1004 to an-master1003. [analytics]
18:43 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71741 and previous config saved to /var/cache/conftool/dbconfig/20241220-184307-root.json [production]
18:35 <kamila@cumin1002> END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1023-1024].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]
18:35 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1024.eqiad.wmnet with OS bookworm [production]
18:28 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71739 and previous config saved to /var/cache/conftool/dbconfig/20241220-182801-root.json [production]
18:15 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1024.eqiad.wmnet with reason: host reimage [production]
18:12 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71738 and previous config saved to /var/cache/conftool/dbconfig/20241220-181256-root.json [production]
18:12 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1024.eqiad.wmnet with reason: host reimage [production]
18:02 <xcollazo@deploy2002> Finished deploy [airflow-dags/analytics@7fecc64]: Pickup hotfix for T377852. (duration: 02m 03s) [production]
18:00 <xcollazo@deploy2002> Started deploy [airflow-dags/analytics@7fecc64]: Pickup hotfix for T377852. [production]
17:57 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71737 and previous config saved to /var/cache/conftool/dbconfig/20241220-175751-root.json [production]
17:53 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1024.eqiad.wmnet with OS bookworm [production]
17:52 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1023.eqiad.wmnet with OS bookworm [production]
17:42 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71736 and previous config saved to /var/cache/conftool/dbconfig/20241220-174246-root.json [production]
17:39 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2168', diff saved to https://phabricator.wikimedia.org/P71735 and previous config saved to /var/cache/conftool/dbconfig/20241220-173922-marostegui.json [production]
17:38 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db2168.codfw.wmnet with reason: maintenance [production]
17:38 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on db2168.codfw.wmnet with reason: maintenance [production]
17:31 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1023.eqiad.wmnet with reason: host reimage [production]
17:26 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1023.eqiad.wmnet with reason: host reimage [production]
17:08 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1023.eqiad.wmnet with OS bookworm [production]
17:06 <kamila@cumin1002> START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1023-1024].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]
16:48 <xcollazo@deploy2002> Finished deploy [airflow-dags/analytics@8c5744d]: Deploying latest analytics Airflow instance DAGs. T377852. (duration: 00m 58s) [production]
16:48 <xcollazo> Deploying latest analytics Airflow instance DAGs. T377852. [analytics]
16:47 <xcollazo@deploy2002> Started deploy [airflow-dags/analytics@8c5744d]: Deploying latest analytics Airflow instance DAGs. T377852. [production]
14:41 <btullis> Started the namenode service on an-master1003 after crash [analytics]
14:38 <btullis@cumin1002> END (FAIL) - Cookbook sre.hadoop.roll-restart-masters (exit_code=99) restart masters for Hadoop analytics cluster: Restart of jvm daemons. [production]
14:06 <btullis@cumin1002> START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop analytics cluster: Restart of jvm daemons. [production]
14:04 <btullis> roll-restarting hadoop nameservers to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/1105893 [analytics]
12:10 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1001-1004].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]
12:10 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1004.eqiad.wmnet with OS bookworm [production]
12:08 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1008-1011].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]
12:08 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1011.eqiad.wmnet with OS bookworm [production]
11:52 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1004.eqiad.wmnet with reason: host reimage [production]
11:48 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1011.eqiad.wmnet with reason: host reimage [production]