251-300 of 10000 results (102ms)
2025-02-06 ยง
17:18 <marostegui@cumin1002> dbctl commit (dc=all): 'db1235 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73334 and previous config saved to /var/cache/conftool/dbconfig/20250206-171835-root.json [production]
17:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db2188 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73333 and previous config saved to /var/cache/conftool/dbconfig/20250206-171626-root.json [production]
17:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db1191 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73332 and previous config saved to /var/cache/conftool/dbconfig/20250206-171601-root.json [production]
17:15 <swfrench-wmf> mw-api-int mw-jobrunner mw-parsoid reverted to 100% PHP 7.4 as of 17:03 - T383845 [production]
17:03 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:03 <marostegui@cumin1002> dbctl commit (dc=all): 'db1235 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73331 and previous config saved to /var/cache/conftool/dbconfig/20250206-170330-root.json [production]
17:03 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:03 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:03 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:02 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply [production]
17:02 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply [production]
17:02 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply [production]
17:01 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply [production]
17:01 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
17:01 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:01 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
17:01 <marostegui@cumin1002> dbctl commit (dc=all): 'db2188 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73330 and previous config saved to /var/cache/conftool/dbconfig/20250206-170121-root.json [production]
17:01 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
17:00 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
17:00 <marostegui@cumin1002> dbctl commit (dc=all): 'db1191 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73329 and previous config saved to /var/cache/conftool/dbconfig/20250206-170055-root.json [production]
16:59 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
16:58 <swfrench@deploy2002> Finished scap sync-world: Backport for [[gerrit:1117931|Disable cookie-based enrollment in 8.1 (T383845)]] (duration: 10m 03s) [production]
16:52 <swfrench@deploy2002> swfrench: Continuing with sync [production]
16:51 <swfrench@deploy2002> swfrench: Backport for [[gerrit:1117931|Disable cookie-based enrollment in 8.1 (T383845)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
16:48 <swfrench@deploy2002> Started scap sync-world: Backport for [[gerrit:1117931|Disable cookie-based enrollment in 8.1 (T383845)]] [production]
16:48 <marostegui@cumin1002> dbctl commit (dc=all): 'db1235 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73328 and previous config saved to /var/cache/conftool/dbconfig/20250206-164825-root.json [production]
16:46 <marostegui@cumin1002> dbctl commit (dc=all): 'db2188 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73327 and previous config saved to /var/cache/conftool/dbconfig/20250206-164615-root.json [production]
16:45 <marostegui@cumin1002> dbctl commit (dc=all): 'db1191 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73326 and previous config saved to /var/cache/conftool/dbconfig/20250206-164550-root.json [production]
16:42 <aikochou@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . [production]
16:34 <aikochou@deploy2002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . [production]
16:33 <marostegui@cumin1002> dbctl commit (dc=all): 'db1235 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73325 and previous config saved to /var/cache/conftool/dbconfig/20250206-163320-root.json [production]
16:31 <marostegui@cumin1002> dbctl commit (dc=all): 'db2188 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73324 and previous config saved to /var/cache/conftool/dbconfig/20250206-163109-root.json [production]
16:30 <marostegui@cumin1002> dbctl commit (dc=all): 'db1191 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73323 and previous config saved to /var/cache/conftool/dbconfig/20250206-163044-root.json [production]
16:18 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [production]
16:18 <marostegui@cumin1002> dbctl commit (dc=all): 'db1235 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73322 and previous config saved to /var/cache/conftool/dbconfig/20250206-161814-root.json [production]
16:18 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/rest-gateway: apply [production]
16:17 <hnowlan@deploy1003> helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [production]
16:17 <hnowlan@deploy1003> helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [production]
16:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db2188 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73321 and previous config saved to /var/cache/conftool/dbconfig/20250206-161604-root.json [production]
16:15 <marostegui@cumin1002> dbctl commit (dc=all): 'db1191 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73320 and previous config saved to /var/cache/conftool/dbconfig/20250206-161539-root.json [production]
16:12 <hnowlan@deploy1003> helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [production]
16:12 <hnowlan@deploy1003> helmfile [staging] START helmfile.d/services/rest-gateway: apply [production]
16:12 <pt1979@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
16:11 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be[2051-2056].codfw.wmnet [production]
16:11 <mvernon@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:11 <mvernon@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-be[2051-2056].codfw.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin2002" [production]
16:09 <mvernon@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-be[2051-2056].codfw.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin2002" [production]
16:05 <pt1979@cumin1002> START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
16:01 <mvernon@cumin2002> START - Cookbook sre.dns.netbox [production]
15:40 <mvernon@cumin2002> START - Cookbook sre.hosts.decommission for hosts ms-be[2051-2056].codfw.wmnet [production]