801-850 of 10000 results (96ms)
2024-09-02 §
09:02 <jayme@cumin1002> START - Cookbook sre.hosts.move-vlan for host wikikube-worker2066 [production]
09:02 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker2066.codfw.wmnet with OS bullseye [production]
09:01 <elukey> restart puppetserver on puppetserver1002 to pick up new JVM settings - T373527 [production]
08:50 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2027 to wikikube-worker2067 [production]
08:49 <jayme@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2067 [production]
08:49 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2008 to wikikube-worker2066 [production]
08:49 <jayme@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2067 [production]
08:49 <jayme@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:48 <jayme@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2066 [production]
08:48 <jayme@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2066 [production]
08:48 <jayme@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:48 <jayme@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2008 to wikikube-worker2066 - jayme@cumin1002" [production]
08:48 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 50%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68515 and previous config saved to /var/cache/conftool/dbconfig/20240902-084833-arnaudb.json [production]
08:48 <jayme@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2008 to wikikube-worker2066 - jayme@cumin1002" [production]
08:47 <jayme@cumin1002> START - Cookbook sre.dns.netbox [production]
08:41 <jayme@cumin1002> START - Cookbook sre.dns.netbox [production]
08:41 <jayme@cumin1002> START - Cookbook sre.hosts.rename from kubernetes2027 to wikikube-worker2067 [production]
08:41 <jayme@cumin1002> START - Cookbook sre.hosts.rename from kubernetes2008 to wikikube-worker2066 [production]
08:33 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 25%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68514 and previous config saved to /var/cache/conftool/dbconfig/20240902-083328-arnaudb.json [production]
08:21 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2027.codfw.wmnet [production]
08:18 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2027.codfw.wmnet [production]
08:18 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2008.codfw.wmnet [production]
08:18 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 15%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68513 and previous config saved to /var/cache/conftool/dbconfig/20240902-081822-arnaudb.json [production]
08:17 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2008.codfw.wmnet [production]
08:09 <moritzm> installing Linux 6.1.106 on Bookworm hosts [production]
08:03 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 6%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68512 and previous config saved to /var/cache/conftool/dbconfig/20240902-080317-arnaudb.json [production]
07:51 <Emperor> restart swift-proxy on ms-fe2012 T360913 [production]
07:48 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 5%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68511 and previous config saved to /var/cache/conftool/dbconfig/20240902-074811-arnaudb.json [production]
07:43 <kartik@deploy1003> Finished scap sync-world: Backport for [[gerrit:1067898|Enable Section Translation in bdr, btm, and dtp Wikpedias (T371420)]] (duration: 07m 35s) [production]
07:39 <kartik@deploy1003> kartik: Continuing with sync [production]
07:38 <kartik@deploy1003> kartik: Backport for [[gerrit:1067898|Enable Section Translation in bdr, btm, and dtp Wikpedias (T371420)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:36 <kartik@deploy1003> Started scap sync-world: Backport for [[gerrit:1067898|Enable Section Translation in bdr, btm, and dtp Wikpedias (T371420)]] [production]
07:33 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 4%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68510 and previous config saved to /var/cache/conftool/dbconfig/20240902-073306-arnaudb.json [production]
07:32 <kartik@deploy1003> Finished scap sync-world: Backport for [[gerrit:1069257|Enable EditCheck references on plwiki (T373079)]] (duration: 28m 41s) [production]
07:22 <kartik@deploy1003> msz2001, kartik: Continuing with sync [production]
07:19 <kartik@deploy1003> msz2001, kartik: Backport for [[gerrit:1069257|Enable EditCheck references on plwiki (T373079)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:18 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 3%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68509 and previous config saved to /var/cache/conftool/dbconfig/20240902-071800-arnaudb.json [production]
07:04 <kartik@deploy1003> Started scap sync-world: Backport for [[gerrit:1069257|Enable EditCheck references on plwiki (T373079)]] [production]
07:02 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 2%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68508 and previous config saved to /var/cache/conftool/dbconfig/20240902-070255-arnaudb.json [production]
06:47 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 1%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68507 and previous config saved to /var/cache/conftool/dbconfig/20240902-064749-arnaudb.json [production]
06:35 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1206.eqiad.wmnet with reason: replag [production]
06:35 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1206.eqiad.wmnet with reason: replag [production]
06:34 <arnaudb@cumin1002> dbctl commit (dc=all): 'depool db1206 replag', diff saved to https://phabricator.wikimedia.org/P68506 and previous config saved to /var/cache/conftool/dbconfig/20240902-063432-arnaudb.json [production]
2024-09-01 §
16:50 <oblivian@cumin1002> dbctl commit (dc=all): 'db1221 (re)pooling @ 100%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68505 and previous config saved to /var/cache/conftool/dbconfig/20240901-165009-root.json [production]
16:35 <oblivian@cumin1002> dbctl commit (dc=all): 'db1221 (re)pooling @ 75%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68504 and previous config saved to /var/cache/conftool/dbconfig/20240901-163504-root.json [production]
16:19 <oblivian@cumin1002> dbctl commit (dc=all): 'db1221 (re)pooling @ 50%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68503 and previous config saved to /var/cache/conftool/dbconfig/20240901-161958-root.json [production]
16:04 <oblivian@cumin1002> dbctl commit (dc=all): 'db1221 (re)pooling @ 25%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68502 and previous config saved to /var/cache/conftool/dbconfig/20240901-160453-root.json [production]
15:50 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1221.eqiad.wmnet [production]
15:50 <mvernon@cumin2002> START - Cookbook sre.hosts.remove-downtime for db1221.eqiad.wmnet [production]
15:49 <oblivian@cumin1002> dbctl commit (dc=all): 'db1221 (re)pooling @ 10%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68501 and previous config saved to /var/cache/conftool/dbconfig/20240901-154948-root.json [production]