2301-2350 of 10000 results (76ms)
2022-11-07 ยง
10:29 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 399338 [production]
10:28 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 59796 [production]
10:28 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 59796 [production]
10:27 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4817 [production]
10:26 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 4817 [production]
10:26 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 35598 [production]
10:26 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 35598 [production]
10:26 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30058 [production]
10:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38218 and previous config saved to /var/cache/conftool/dbconfig/20221107-102555-ladsgroup.json [production]
10:25 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 30058 [production]
10:25 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 46416 [production]
10:25 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 46416 [production]
10:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38217 and previous config saved to /var/cache/conftool/dbconfig/20221107-102509-marostegui.json [production]
10:25 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
10:25 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
10:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38216 and previous config saved to /var/cache/conftool/dbconfig/20221107-102458-marostegui.json [production]
10:24 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3214 [production]
10:24 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 3214 [production]
10:21 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 17511 [production]
10:21 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 17511 [production]
10:20 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 12400 [production]
10:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38215 and previous config saved to /var/cache/conftool/dbconfig/20221107-102043-ladsgroup.json [production]
10:20 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 12400 [production]
10:19 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 7459 [production]
10:19 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 7459 [production]
10:17 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 6661 [production]
10:17 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 6661 [production]
10:17 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 5398 [production]
10:17 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 5398 [production]
10:17 <marostegui@cumin1001> dbctl commit (dc=all): 'es2024 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38214 and previous config saved to /var/cache/conftool/dbconfig/20221107-101700-root.json [production]
10:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38213 and previous config saved to /var/cache/conftool/dbconfig/20221107-101140-marostegui.json [production]
10:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance [production]
10:11 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1146.eqiad.wmnet with reason: Maintenance [production]
10:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
10:11 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
10:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38212 and previous config saved to /var/cache/conftool/dbconfig/20221107-101102-marostegui.json [production]
10:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38211 and previous config saved to /var/cache/conftool/dbconfig/20221107-101048-ladsgroup.json [production]
10:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38210 and previous config saved to /var/cache/conftool/dbconfig/20221107-100952-marostegui.json [production]
10:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P38209 and previous config saved to /var/cache/conftool/dbconfig/20221107-100536-ladsgroup.json [production]
10:01 <marostegui@cumin1001> dbctl commit (dc=all): 'es2024 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38208 and previous config saved to /var/cache/conftool/dbconfig/20221107-100155-root.json [production]
09:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P38207 and previous config saved to /var/cache/conftool/dbconfig/20221107-095556-marostegui.json [production]
09:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38206 and previous config saved to /var/cache/conftool/dbconfig/20221107-095542-ladsgroup.json [production]
09:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P38205 and previous config saved to /var/cache/conftool/dbconfig/20221107-095445-marostegui.json [production]
09:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38204 and previous config saved to /var/cache/conftool/dbconfig/20221107-095149-ladsgroup.json [production]
09:51 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
09:51 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
09:51 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
09:51 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
09:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38203 and previous config saved to /var/cache/conftool/dbconfig/20221107-095030-ladsgroup.json [production]
09:49 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]