101-150 of 10000 results (26ms)
2024-12-23 §
08:02 <akosiaris@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-ctrl1004 [production]
08:02 <akosiaris@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-ctrl1004 [production]
08:02 <akosiaris@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:02 <akosiaris@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wikikube-worker1290 to wikikube-ctrl1004 - akosiaris@cumin1002" [production]
07:57 <akosiaris@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wikikube-worker1290 to wikikube-ctrl1004 - akosiaris@cumin1002" [production]
07:53 <akosiaris@cumin1002> START - Cookbook sre.dns.netbox [production]
07:53 <akosiaris@cumin1002> START - Cookbook sre.hosts.rename from wikikube-worker1290 to wikikube-ctrl1004 [production]
07:46 <akosiaris@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1290.eqiad.wmnet [production]
07:46 <akosiaris@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1290.eqiad.wmnet [production]
07:03 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on dbproxy1029.eqiad.wmnet with reason: maintenance [production]
07:03 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 10 days, 0:00:00 on dbproxy1029.eqiad.wmnet with reason: maintenance [production]
07:03 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on dbproxy1028.eqiad.wmnet with reason: maintenance [production]
07:03 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 10 days, 0:00:00 on dbproxy1028.eqiad.wmnet with reason: maintenance [production]
2024-12-22 §
11:43 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [production]
11:40 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [production]
10:52 <Emperor> restart swift-object on ms-be2082 [production]
10:49 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [production]
10:46 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [production]
2024-12-21 §
22:52 <wmbot~lucaswerkmeister@tools-bastion-13> deployed eb9d0ae3c2 (l10n updates: lb, pa; also upgrade dependencies, including Flask 3.1.0 and Jinja2 3.1.5) [tools.lexeme-forms]
00:00 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
00:00 <andrew@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002" [production]
2024-12-20 §
23:01 <brennen> brennen@releases1003:/srv/org/wikimedia/releases/mediawiki/1.43$ sudo cp -i ~atieno/mediawiki-* ./ [releng]
22:12 <andrew@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002" [production]
21:52 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd2004-dev.codfw.wmnet with reason: host reimage [production]
21:48 <andrew@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd2004-dev.codfw.wmnet with reason: host reimage [production]
21:29 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
21:24 <andrew@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
21:13 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm [production]
19:46 <btullis> restarted the hadoop-hdfs-namenode service on an-master1004 to pick up the new settins as well. [analytics]
19:46 <btullis> failed over the hadoop namenode services from an-master1004 to an-master1003. [analytics]
18:43 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71741 and previous config saved to /var/cache/conftool/dbconfig/20241220-184307-root.json [production]
18:35 <kamila@cumin1002> END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1023-1024].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]
18:35 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1024.eqiad.wmnet with OS bookworm [production]
18:28 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71739 and previous config saved to /var/cache/conftool/dbconfig/20241220-182801-root.json [production]
18:15 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1024.eqiad.wmnet with reason: host reimage [production]
18:12 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71738 and previous config saved to /var/cache/conftool/dbconfig/20241220-181256-root.json [production]
18:12 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1024.eqiad.wmnet with reason: host reimage [production]
18:02 <xcollazo@deploy2002> Finished deploy [airflow-dags/analytics@7fecc64]: Pickup hotfix for T377852. (duration: 02m 03s) [production]
18:00 <xcollazo@deploy2002> Started deploy [airflow-dags/analytics@7fecc64]: Pickup hotfix for T377852. [production]
17:57 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71737 and previous config saved to /var/cache/conftool/dbconfig/20241220-175751-root.json [production]
17:53 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1024.eqiad.wmnet with OS bookworm [production]
17:52 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1023.eqiad.wmnet with OS bookworm [production]
17:42 <marostegui@cumin1002> dbctl commit (dc=all): 'db2168 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71736 and previous config saved to /var/cache/conftool/dbconfig/20241220-174246-root.json [production]
17:39 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2168', diff saved to https://phabricator.wikimedia.org/P71735 and previous config saved to /var/cache/conftool/dbconfig/20241220-173922-marostegui.json [production]
17:38 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db2168.codfw.wmnet with reason: maintenance [production]
17:38 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on db2168.codfw.wmnet with reason: maintenance [production]
17:31 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1023.eqiad.wmnet with reason: host reimage [production]
17:26 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1023.eqiad.wmnet with reason: host reimage [production]
17:08 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1023.eqiad.wmnet with OS bookworm [production]
17:06 <kamila@cumin1002> START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1023-1024].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [production]