1351-1400 of 10000 results (74ms)
2022-10-31 ยง
18:00 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
18:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318605)', diff saved to https://phabricator.wikimedia.org/P37254 and previous config saved to /var/cache/conftool/dbconfig/20221031-180021-ladsgroup.json [production]
17:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1202.eqiad.wmnet with reason: Maintenance [production]
17:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318955)', diff saved to https://phabricator.wikimedia.org/P37250 and previous config saved to /var/cache/conftool/dbconfig/20221031-174301-ladsgroup.json [production]
17:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37249 and previous config saved to /var/cache/conftool/dbconfig/20221031-174052-ladsgroup.json [production]
17:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P37248 and previous config saved to /var/cache/conftool/dbconfig/20221031-173008-ladsgroup.json [production]
17:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P37247 and previous config saved to /var/cache/conftool/dbconfig/20221031-172755-ladsgroup.json [production]
17:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P37246 and previous config saved to /var/cache/conftool/dbconfig/20221031-172545-ladsgroup.json [production]
17:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T318605)', diff saved to https://phabricator.wikimedia.org/P37245 and previous config saved to /var/cache/conftool/dbconfig/20221031-171501-ladsgroup.json [production]
17:14 <mutante> contint1001 - racadm serveraction powercyle - crashed [production]
17:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P37244 and previous config saved to /var/cache/conftool/dbconfig/20221031-171248-ladsgroup.json [production]
17:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P37243 and previous config saved to /var/cache/conftool/dbconfig/20221031-171039-ladsgroup.json [production]
17:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1170:3312 (T318605)', diff saved to https://phabricator.wikimedia.org/P37242 and previous config saved to /var/cache/conftool/dbconfig/20221031-170935-ladsgroup.json [production]
17:09 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
17:09 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
17:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37241 and previous config saved to /var/cache/conftool/dbconfig/20221031-170925-ladsgroup.json [production]
17:02 <mutante> contint1001 - just went fully down without maintenance work, fortunately 2001 is the prod CI server currently [production]
16:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318955)', diff saved to https://phabricator.wikimedia.org/P37240 and previous config saved to /var/cache/conftool/dbconfig/20221031-165742-ladsgroup.json [production]
16:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37239 and previous config saved to /var/cache/conftool/dbconfig/20221031-165532-ladsgroup.json [production]
16:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1194 (T318955)', diff saved to https://phabricator.wikimedia.org/P37238 and previous config saved to /var/cache/conftool/dbconfig/20221031-165532-ladsgroup.json [production]
16:55 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1194.eqiad.wmnet with reason: Maintenance [production]
16:55 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1194.eqiad.wmnet with reason: Maintenance [production]
16:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318955)', diff saved to https://phabricator.wikimedia.org/P37237 and previous config saved to /var/cache/conftool/dbconfig/20221031-165511-ladsgroup.json [production]
16:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37236 and previous config saved to /var/cache/conftool/dbconfig/20221031-165418-ladsgroup.json [production]
16:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37235 and previous config saved to /var/cache/conftool/dbconfig/20221031-164431-ladsgroup.json [production]
16:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2182.codfw.wmnet with reason: Maintenance [production]
16:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2182.codfw.wmnet with reason: Maintenance [production]
16:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318955)', diff saved to https://phabricator.wikimedia.org/P37234 and previous config saved to /var/cache/conftool/dbconfig/20221031-164409-ladsgroup.json [production]
16:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P37233 and previous config saved to /var/cache/conftool/dbconfig/20221031-164004-ladsgroup.json [production]
16:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37232 and previous config saved to /var/cache/conftool/dbconfig/20221031-163912-ladsgroup.json [production]
16:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P37231 and previous config saved to /var/cache/conftool/dbconfig/20221031-162903-ladsgroup.json [production]
16:25 <hashar@deploy1002> Finished deploy [integration/docroot@0ff8642]: build: Use disableProcessTimeout() for serve commands only (duration: 00m 25s) [production]
16:24 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P37230 and previous config saved to /var/cache/conftool/dbconfig/20221031-162458-ladsgroup.json [production]
16:24 <hashar@deploy1002> Started deploy [integration/docroot@0ff8642]: build: Use disableProcessTimeout() for serve commands only [production]
16:24 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37229 and previous config saved to /var/cache/conftool/dbconfig/20221031-162405-ladsgroup.json [production]
16:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2170:3312 (T318605)', diff saved to https://phabricator.wikimedia.org/P37228 and previous config saved to /var/cache/conftool/dbconfig/20221031-162311-ladsgroup.json [production]
16:23 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
16:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
16:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318605)', diff saved to https://phabricator.wikimedia.org/P37227 and previous config saved to /var/cache/conftool/dbconfig/20221031-162249-ladsgroup.json [production]
16:17 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe [production]
16:17 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-be [production]
16:17 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-tls [production]
16:15 <aborrero@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37226 and previous config saved to /var/cache/conftool/dbconfig/20221031-161448-ladsgroup.json [production]
16:14 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
16:14 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
16:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318605)', diff saved to https://phabricator.wikimedia.org/P37225 and previous config saved to /var/cache/conftool/dbconfig/20221031-161426-ladsgroup.json [production]
16:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P37224 and previous config saved to /var/cache/conftool/dbconfig/20221031-161356-ladsgroup.json [production]
16:13 <aborrero@cumin1001> START - Cookbook sre.dns.netbox [production]
16:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318955)', diff saved to https://phabricator.wikimedia.org/P37223 and previous config saved to /var/cache/conftool/dbconfig/20221031-160951-ladsgroup.json [production]