801-850 of 10000 results (105ms)
2024-07-10 ยง
19:03 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1193 (T367781)', diff saved to https://phabricator.wikimedia.org/P66172 and previous config saved to /var/cache/conftool/dbconfig/20240710-190352-arnaudb.json [production]
19:03 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1193 (T367781)', diff saved to https://phabricator.wikimedia.org/P66171 and previous config saved to /var/cache/conftool/dbconfig/20240710-190244-arnaudb.json [production]
19:02 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1193.eqiad.wmnet with reason: Maintenance [production]
19:02 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1193.eqiad.wmnet with reason: Maintenance [production]
19:02 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T367781)', diff saved to https://phabricator.wikimedia.org/P66170 and previous config saved to /var/cache/conftool/dbconfig/20240710-190222-arnaudb.json [production]
18:56 <sfaci@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply [production]
18:56 <sfaci@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply [production]
18:47 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P66169 and previous config saved to /var/cache/conftool/dbconfig/20240710-184714-arnaudb.json [production]
18:45 <ryankemper@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:45 <ryankemper@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add 4 new IPs (2 eqiad, 2 codfw) for wdqs graph split - ryankemper@cumin2002" [production]
18:43 <ryankemper@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add 4 new IPs (2 eqiad, 2 codfw) for wdqs graph split - ryankemper@cumin2002" [production]
18:35 <ryankemper@cumin2002> START - Cookbook sre.dns.netbox [production]
18:32 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P66168 and previous config saved to /var/cache/conftool/dbconfig/20240710-183207-arnaudb.json [production]
18:17 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T367781)', diff saved to https://phabricator.wikimedia.org/P66166 and previous config saved to /var/cache/conftool/dbconfig/20240710-181700-arnaudb.json [production]
17:16 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1192 (T367781)', diff saved to https://phabricator.wikimedia.org/P66164 and previous config saved to /var/cache/conftool/dbconfig/20240710-171644-arnaudb.json [production]
17:16 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
17:16 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
17:16 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T367781)', diff saved to https://phabricator.wikimedia.org/P66163 and previous config saved to /var/cache/conftool/dbconfig/20240710-171622-arnaudb.json [production]
17:01 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 100%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66162 and previous config saved to /var/cache/conftool/dbconfig/20240710-170143-arnaudb.json [production]
17:01 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P66161 and previous config saved to /var/cache/conftool/dbconfig/20240710-170115-arnaudb.json [production]
16:46 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 75%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66160 and previous config saved to /var/cache/conftool/dbconfig/20240710-164637-arnaudb.json [production]
16:46 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P66159 and previous config saved to /var/cache/conftool/dbconfig/20240710-164608-arnaudb.json [production]
16:42 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T352010)', diff saved to https://phabricator.wikimedia.org/P66158 and previous config saved to /var/cache/conftool/dbconfig/20240710-164225-ladsgroup.json [production]
16:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 50%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66157 and previous config saved to /var/cache/conftool/dbconfig/20240710-163131-arnaudb.json [production]
16:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T367781)', diff saved to https://phabricator.wikimedia.org/P66156 and previous config saved to /var/cache/conftool/dbconfig/20240710-163100-arnaudb.json [production]
16:30 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1178 (T367781)', diff saved to https://phabricator.wikimedia.org/P66155 and previous config saved to /var/cache/conftool/dbconfig/20240710-162952-arnaudb.json [production]
16:29 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
16:29 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
16:29 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T367781)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20240710-162926-arnaudb.json [production]
16:27 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P66153 and previous config saved to /var/cache/conftool/dbconfig/20240710-162718-ladsgroup.json [production]
16:17 <swfrench@deploy1002> helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [production]
16:17 <swfrench@deploy1002> helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [production]
16:16 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 25%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66152 and previous config saved to /var/cache/conftool/dbconfig/20240710-161626-arnaudb.json [production]
16:14 <brett@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary (T368083) [production]
16:14 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P66151 and previous config saved to /var/cache/conftool/dbconfig/20240710-161419-arnaudb.json [production]
16:12 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P66150 and previous config saved to /var/cache/conftool/dbconfig/20240710-161211-ladsgroup.json [production]
16:11 <brett@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary (T368083) [production]
16:08 <brett@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-high-traffic2 (T368083) [production]
16:05 <brett@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-high-traffic2 (T368083) [production]
16:02 <brett@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-high-traffic1 (T368083) [production]
16:01 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 10%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66149 and previous config saved to /var/cache/conftool/dbconfig/20240710-160120-arnaudb.json [production]
16:01 <swfrench@deploy1002> helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [production]
16:00 <swfrench@deploy1002> helmfile [codfw] START helmfile.d/services/rest-gateway: apply [production]
16:00 <brett@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-high-traffic1 (T368083) [production]
15:59 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P66148 and previous config saved to /var/cache/conftool/dbconfig/20240710-155911-arnaudb.json [production]
15:59 <swfrench@deploy1002> helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [production]
15:58 <swfrench@deploy1002> helmfile [staging] START helmfile.d/services/rest-gateway: apply [production]
15:57 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T352010)', diff saved to https://phabricator.wikimedia.org/P66147 and previous config saved to /var/cache/conftool/dbconfig/20240710-155703-ladsgroup.json [production]
15:55 <brett@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-high-traffic2-eqsin (T368083) [production]
15:54 <brett@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-high-traffic2-eqsin (T368083) [production]