2101-2150 of 10000 results (81ms)
2024-02-06 §
06:02 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance [production]
06:01 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host es1029.eqiad.wmnet with OS bookworm [production]
05:58 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1029 T351916', diff saved to https://phabricator.wikimedia.org/P56283 and previous config saved to /var/cache/conftool/dbconfig/20240206-055835-root.json [production]
04:55 <mwpresync@deploy2002> Finished scap: testwikis wikis to 1.42.0-wmf.17 refs T354435 (duration: 51m 02s) [production]
04:04 <mwpresync@deploy2002> Started scap: testwikis wikis to 1.42.0-wmf.17 refs T354435 [production]
04:02 <mwpresync@deploy2002> Pruned MediaWiki: 1.42.0-wmf.14 (duration: 02m 07s) [production]
03:00 <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[2030-2035].codfw.wmnet: Apply updated JVM — T356648 - eevans@cumin1002 [production]
02:03 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2030-2035].codfw.wmnet: Apply updated JVM — T356648 - eevans@cumin1002 [production]
02:00 <eevans@cumin1002> END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching restbase[2026-2035].codfw.wmnet: Apply updated JVM — T356648 - eevans@cumin1002 [production]
01:19 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2026-2035].codfw.wmnet: Apply updated JVM — T356648 - eevans@cumin1002 [production]
01:18 <eevans@cumin1002> END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching restbase[2021-2035].codfw.wmnet: Apply updated JVM — T356648 - eevans@cumin1002 [production]
00:29 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2021-2035].codfw.wmnet: Apply updated JVM — T356648 - eevans@cumin1002 [production]
00:26 <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[1028-1033].eqiad.wmnet: Apply updated JVM — T356648 - eevans@cumin1002 [production]
2024-02-05 §
23:39 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply cluster settings before private IP migration - bking@cumin2002 - T355617 [production]
23:31 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[1028-1033].eqiad.wmnet: Apply updated JVM — T356648 - eevans@cumin1002 [production]
23:19 <eevans@cumin1002> END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching A:restbase-eqiad: Apply updated JVM — T356648 - eevans@cumin1002 [production]
23:12 <zabe@deploy2002> Finished scap: Backport for [[gerrit:997281|MobileFrontend hook should not apply outside mobile view (T356711)]] (duration: 09m 18s) [production]
23:06 <zabe@deploy2002> zabe: Continuing with sync [production]
23:04 <zabe@deploy2002> zabe: Backport for [[gerrit:997281|MobileFrontend hook should not apply outside mobile view (T356711)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
23:03 <zabe@deploy2002> Started scap: Backport for [[gerrit:997281|MobileFrontend hook should not apply outside mobile view (T356711)]] [production]
22:30 <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply cluster settings before private IP migration - bking@cumin2002 - T355617 [production]
22:29 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cloudelastic1009.wikimedia.org with reason: T355617 [production]
22:28 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cloudelastic1009.wikimedia.org with reason: T355617 [production]
22:25 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cloudelastic1009.wikimedia.org for migrate cloudelastic1009 to private IP - bking@cumin2002 - T355617 [production]
22:25 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1009.wikimedia.org for migrate cloudelastic1009 to private IP - bking@cumin2002 - T355617 [production]
22:25 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2192 (T355609)', diff saved to https://phabricator.wikimedia.org/P56282 and previous config saved to /var/cache/conftool/dbconfig/20240205-222507-marostegui.json [production]
22:23 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on sessionstore1003.eqiad.wmnet with reason: Decommissioning — T353405 [production]
22:23 <bking@cumin2002> conftool action : set/pooled=no; selector: name=cloudelastic1009.wikimedia.org [production]
22:23 <eevans@cumin1002> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on sessionstore1003.eqiad.wmnet with reason: Decommissioning — T353405 [production]
22:23 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on sessionstore1002.eqiad.wmnet with reason: Decommissioning — T353405 [production]
22:23 <eevans@cumin1002> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on sessionstore1002.eqiad.wmnet with reason: Decommissioning — T353405 [production]
22:23 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on sessionstore1001.eqiad.wmnet with reason: Decommissioning — T353405 [production]
22:22 <eevans@cumin1002> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on sessionstore1001.eqiad.wmnet with reason: Decommissioning — T353405 [production]
22:10 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P56281 and previous config saved to /var/cache/conftool/dbconfig/20240205-221001-marostegui.json [production]
22:06 <brett@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ncmonitor1001.eqiad.wmnet - brett@cumin2002" [production]
22:05 <urandom> Decommissioning Cassandra, sessionstore1001 — T353405 [production]
22:05 <brett@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ncmonitor1001.eqiad.wmnet - brett@cumin2002" [production]
22:04 <brett@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ncmonitor1001.eqiad.wmnet on all recursors [production]
22:04 <brett@cumin2002> START - Cookbook sre.dns.wipe-cache ncmonitor1001.eqiad.wmnet on all recursors [production]
22:04 <brett@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:04 <brett@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ncmonitor1001.eqiad.wmnet - brett@cumin2002" [production]
22:03 <brett@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ncmonitor1001.eqiad.wmnet - brett@cumin2002" [production]
22:01 <brett@cumin2002> START - Cookbook sre.dns.netbox [production]
22:01 <brett@cumin2002> START - Cookbook sre.ganeti.makevm for new host ncmonitor1001.eqiad.wmnet [production]
21:58 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Apply updated JVM — T356648 - eevans@cumin1002 [production]
21:56 <cjming> end of UTC late backport window [production]
21:56 <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore[1001-1006].eqiad.wmnet: Apply updated JVM — T356648 - eevans@cumin1002 [production]
21:55 <cjming@deploy2002> Finished scap: Backport for [[gerrit:997279|Use decodeURI for comment ID searches as well as heading searches (T356199)]] (duration: 13m 08s) [production]
21:54 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P56280 and previous config saved to /var/cache/conftool/dbconfig/20240205-215454-marostegui.json [production]
21:49 <cjming@deploy2002> cjming and kemayo: Continuing with sync [production]