1351-1400 of 10000 results (44ms)
2022-03-08 ยง
13:07 <aborrero@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1005.wikimedia.org [production]
13:01 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P22096 and previous config saved to /var/cache/conftool/dbconfig/20220308-130145-marostegui.json [production]
12:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1174 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P22095 and previous config saved to /var/cache/conftool/dbconfig/20220308-125806-root.json [production]
12:57 <aborrero@cumin2002> START - Cookbook sre.hosts.reboot-single for host cloudcontrol1005.wikimedia.org [production]
12:56 <aborrero@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1021.eqiad.wmnet [production]
12:51 <aborrero@cumin2002> START - Cookbook sre.hosts.reboot-single for host cloudcephosd1021.eqiad.wmnet [production]
12:51 <aborrero@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudcephosd1021.eqiad.wmnet [production]
12:51 <aborrero@cumin2002> START - Cookbook sre.hosts.reboot-single for host cloudcephosd1021.eqiad.wmnet [production]
12:46 <btullis@cumin2002> START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons. [production]
12:46 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298294)', diff saved to https://phabricator.wikimedia.org/P22094 and previous config saved to /var/cache/conftool/dbconfig/20220308-124610-marostegui.json [production]
12:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1174 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P22093 and previous config saved to /var/cache/conftool/dbconfig/20220308-124302-root.json [production]
12:42 <marostegui@cumin2002> dbctl commit (dc=all): 'Depooling db1096:3315 (T298294)', diff saved to https://phabricator.wikimedia.org/P22092 and previous config saved to /var/cache/conftool/dbconfig/20220308-124257-marostegui.json [production]
12:42 <btullis@cumin2002> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid public cluster: Roll restart of Druid jvm daemons. [production]
12:42 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
12:42 <marostegui@cumin2002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
12:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1174 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P22091 and previous config saved to /var/cache/conftool/dbconfig/20220308-122752-root.json [production]
12:16 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on 8 hosts with reason: Maintenance [production]
12:16 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 16:00:00 on 8 hosts with reason: Maintenance [production]
12:16 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
12:16 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
12:15 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
12:15 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
12:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
12:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
12:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298294)', diff saved to https://phabricator.wikimedia.org/P22090 and previous config saved to /var/cache/conftool/dbconfig/20220308-121443-marostegui.json [production]
12:00 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
11:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P22089 and previous config saved to /var/cache/conftool/dbconfig/20220308-115938-marostegui.json [production]
11:59 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
11:59 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
11:58 <volans> uploaded spicerack_2.2.0 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
11:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
11:55 <reedy@deploy1002> Synchronized wmf-config/CommonSettings.php: Use namespaced ApiFeatureUsageQueryEngineElastica T302907 (duration: 00m 49s) [production]
11:54 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 6 hosts with reason: Maintenance [production]
11:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 6 hosts with reason: Maintenance [production]
11:54 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
11:53 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
11:51 <btullis@cumin2002> START - Cookbook sre.druid.roll-restart-workers for Druid public cluster: Roll restart of Druid jvm daemons. [production]
11:50 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1083.eqiad.wmnet with OS buster [production]
11:48 <vgutierrez> pool cp1083 with HAProxy as TLS termination layer - T290005 [production]
11:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P22088 and previous config saved to /var/cache/conftool/dbconfig/20220308-114434-marostegui.json [production]
11:41 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
11:41 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
11:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 100%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P22086 and previous config saved to /var/cache/conftool/dbconfig/20220308-113424-root.json [production]
11:31 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2008.codfw.wmnet [production]
11:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1123 (T300381)', diff saved to https://phabricator.wikimedia.org/P22085 and previous config saved to /var/cache/conftool/dbconfig/20220308-113110-marostegui.json [production]
11:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
11:31 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
11:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T300381)', diff saved to https://phabricator.wikimedia.org/P22084 and previous config saved to /var/cache/conftool/dbconfig/20220308-113102-marostegui.json [production]
11:30 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage [production]
11:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298294)', diff saved to https://phabricator.wikimedia.org/P22083 and previous config saved to /var/cache/conftool/dbconfig/20220308-112929-marostegui.json [production]