|
2022-10-31
ยง
|
| 17:09 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37241 and previous config saved to /var/cache/conftool/dbconfig/20221031-170925-ladsgroup.json |
[production] |
| 17:02 |
<mutante> |
contint1001 - just went fully down without maintenance work, fortunately 2001 is the prod CI server currently |
[production] |
| 16:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318955)', diff saved to https://phabricator.wikimedia.org/P37240 and previous config saved to /var/cache/conftool/dbconfig/20221031-165742-ladsgroup.json |
[production] |
| 16:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37239 and previous config saved to /var/cache/conftool/dbconfig/20221031-165532-ladsgroup.json |
[production] |
| 16:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1194 (T318955)', diff saved to https://phabricator.wikimedia.org/P37238 and previous config saved to /var/cache/conftool/dbconfig/20221031-165532-ladsgroup.json |
[production] |
| 16:55 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1194.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:55 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1194.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318955)', diff saved to https://phabricator.wikimedia.org/P37237 and previous config saved to /var/cache/conftool/dbconfig/20221031-165511-ladsgroup.json |
[production] |
| 16:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37236 and previous config saved to /var/cache/conftool/dbconfig/20221031-165418-ladsgroup.json |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2182 (T318955)', diff saved to https://phabricator.wikimedia.org/P37235 and previous config saved to /var/cache/conftool/dbconfig/20221031-164431-ladsgroup.json |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2182.codfw.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2182.codfw.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318955)', diff saved to https://phabricator.wikimedia.org/P37234 and previous config saved to /var/cache/conftool/dbconfig/20221031-164409-ladsgroup.json |
[production] |
| 16:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P37233 and previous config saved to /var/cache/conftool/dbconfig/20221031-164004-ladsgroup.json |
[production] |
| 16:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37232 and previous config saved to /var/cache/conftool/dbconfig/20221031-163912-ladsgroup.json |
[production] |
| 16:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P37231 and previous config saved to /var/cache/conftool/dbconfig/20221031-162903-ladsgroup.json |
[production] |
| 16:25 |
<hashar@deploy1002> |
Finished deploy [integration/docroot@0ff8642]: build: Use disableProcessTimeout() for serve commands only (duration: 00m 25s) |
[production] |
| 16:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P37230 and previous config saved to /var/cache/conftool/dbconfig/20221031-162458-ladsgroup.json |
[production] |
| 16:24 |
<hashar@deploy1002> |
Started deploy [integration/docroot@0ff8642]: build: Use disableProcessTimeout() for serve commands only |
[production] |
| 16:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37229 and previous config saved to /var/cache/conftool/dbconfig/20221031-162405-ladsgroup.json |
[production] |
| 16:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2170:3312 (T318605)', diff saved to https://phabricator.wikimedia.org/P37228 and previous config saved to /var/cache/conftool/dbconfig/20221031-162311-ladsgroup.json |
[production] |
| 16:23 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance |
[production] |
| 16:22 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance |
[production] |
| 16:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318605)', diff saved to https://phabricator.wikimedia.org/P37227 and previous config saved to /var/cache/conftool/dbconfig/20221031-162249-ladsgroup.json |
[production] |
| 16:17 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe |
[production] |
| 16:17 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-be |
[production] |
| 16:17 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-tls |
[production] |
| 16:15 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37226 and previous config saved to /var/cache/conftool/dbconfig/20221031-161448-ladsgroup.json |
[production] |
| 16:14 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:14 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318605)', diff saved to https://phabricator.wikimedia.org/P37225 and previous config saved to /var/cache/conftool/dbconfig/20221031-161426-ladsgroup.json |
[production] |
| 16:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P37224 and previous config saved to /var/cache/conftool/dbconfig/20221031-161356-ladsgroup.json |
[production] |
| 16:13 |
<aborrero@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318955)', diff saved to https://phabricator.wikimedia.org/P37223 and previous config saved to /var/cache/conftool/dbconfig/20221031-160951-ladsgroup.json |
[production] |
| 16:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37222 and previous config saved to /var/cache/conftool/dbconfig/20221031-160743-ladsgroup.json |
[production] |
| 16:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1191 (T318955)', diff saved to https://phabricator.wikimedia.org/P37221 and previous config saved to /var/cache/conftool/dbconfig/20221031-160641-ladsgroup.json |
[production] |
| 16:06 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1191.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:06 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1191.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T318955)', diff saved to https://phabricator.wikimedia.org/P37220 and previous config saved to /var/cache/conftool/dbconfig/20221031-160620-ladsgroup.json |
[production] |
| 15:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37219 and previous config saved to /var/cache/conftool/dbconfig/20221031-155919-ladsgroup.json |
[production] |
| 15:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318955)', diff saved to https://phabricator.wikimedia.org/P37218 and previous config saved to /var/cache/conftool/dbconfig/20221031-155850-ladsgroup.json |
[production] |
| 15:56 |
<ryankemper> |
[Elastic] `ryankemper@elastic2052:~$ sudo reboot` to grab latest kernel |
[production] |
| 15:54 |
<ryankemper> |
[Elastic] `ryankemper@elastic2043:~$ sudo pool` (cluster back to green and DIMM A2 has been switched out by dc-ops); marked as `Active` in netbox |
[production] |
| 15:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.drmrs.wmnet,service=varnish-fe |
[production] |
| 15:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.drmrs.wmnet,service=ats-be |
[production] |
| 15:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.drmrs.wmnet,service=ats-tls |
[production] |
| 15:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37217 and previous config saved to /var/cache/conftool/dbconfig/20221031-155236-ladsgroup.json |
[production] |
| 15:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P37216 and previous config saved to /var/cache/conftool/dbconfig/20221031-155113-ladsgroup.json |
[production] |