|
2022-10-31
ยง
|
| 16:44 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2182.codfw.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318955)', diff saved to https://phabricator.wikimedia.org/P37234 and previous config saved to /var/cache/conftool/dbconfig/20221031-164409-ladsgroup.json |
[production] |
| 16:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P37233 and previous config saved to /var/cache/conftool/dbconfig/20221031-164004-ladsgroup.json |
[production] |
| 16:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P37232 and previous config saved to /var/cache/conftool/dbconfig/20221031-163912-ladsgroup.json |
[production] |
| 16:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P37231 and previous config saved to /var/cache/conftool/dbconfig/20221031-162903-ladsgroup.json |
[production] |
| 16:25 |
<hashar@deploy1002> |
Finished deploy [integration/docroot@0ff8642]: build: Use disableProcessTimeout() for serve commands only (duration: 00m 25s) |
[production] |
| 16:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P37230 and previous config saved to /var/cache/conftool/dbconfig/20221031-162458-ladsgroup.json |
[production] |
| 16:24 |
<hashar@deploy1002> |
Started deploy [integration/docroot@0ff8642]: build: Use disableProcessTimeout() for serve commands only |
[production] |
| 16:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37229 and previous config saved to /var/cache/conftool/dbconfig/20221031-162405-ladsgroup.json |
[production] |
| 16:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2170:3312 (T318605)', diff saved to https://phabricator.wikimedia.org/P37228 and previous config saved to /var/cache/conftool/dbconfig/20221031-162311-ladsgroup.json |
[production] |
| 16:23 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance |
[production] |
| 16:22 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance |
[production] |
| 16:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2148 (T318605)', diff saved to https://phabricator.wikimedia.org/P37227 and previous config saved to /var/cache/conftool/dbconfig/20221031-162249-ladsgroup.json |
[production] |
| 16:17 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=varnish-fe |
[production] |
| 16:17 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-be |
[production] |
| 16:17 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=ats-tls |
[production] |
| 16:15 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1162 (T318605)', diff saved to https://phabricator.wikimedia.org/P37226 and previous config saved to /var/cache/conftool/dbconfig/20221031-161448-ladsgroup.json |
[production] |
| 16:14 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:14 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1156 (T318605)', diff saved to https://phabricator.wikimedia.org/P37225 and previous config saved to /var/cache/conftool/dbconfig/20221031-161426-ladsgroup.json |
[production] |
| 16:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P37224 and previous config saved to /var/cache/conftool/dbconfig/20221031-161356-ladsgroup.json |
[production] |
| 16:13 |
<aborrero@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318955)', diff saved to https://phabricator.wikimedia.org/P37223 and previous config saved to /var/cache/conftool/dbconfig/20221031-160951-ladsgroup.json |
[production] |
| 16:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37222 and previous config saved to /var/cache/conftool/dbconfig/20221031-160743-ladsgroup.json |
[production] |
| 16:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1191 (T318955)', diff saved to https://phabricator.wikimedia.org/P37221 and previous config saved to /var/cache/conftool/dbconfig/20221031-160641-ladsgroup.json |
[production] |
| 16:06 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1191.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:06 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1191.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T318955)', diff saved to https://phabricator.wikimedia.org/P37220 and previous config saved to /var/cache/conftool/dbconfig/20221031-160620-ladsgroup.json |
[production] |
| 15:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37219 and previous config saved to /var/cache/conftool/dbconfig/20221031-155919-ladsgroup.json |
[production] |
| 15:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T318955)', diff saved to https://phabricator.wikimedia.org/P37218 and previous config saved to /var/cache/conftool/dbconfig/20221031-155850-ladsgroup.json |
[production] |
| 15:56 |
<ryankemper> |
[Elastic] `ryankemper@elastic2052:~$ sudo reboot` to grab latest kernel |
[production] |
| 15:54 |
<ryankemper> |
[Elastic] `ryankemper@elastic2043:~$ sudo pool` (cluster back to green and DIMM A2 has been switched out by dc-ops); marked as `Active` in netbox |
[production] |
| 15:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.drmrs.wmnet,service=varnish-fe |
[production] |
| 15:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.drmrs.wmnet,service=ats-be |
[production] |
| 15:53 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.drmrs.wmnet,service=ats-tls |
[production] |
| 15:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P37217 and previous config saved to /var/cache/conftool/dbconfig/20221031-155236-ladsgroup.json |
[production] |
| 15:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P37216 and previous config saved to /var/cache/conftool/dbconfig/20221031-155113-ladsgroup.json |
[production] |
| 15:51 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.drmrs.wmnet,service=varnish-fe |
[production] |
| 15:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.drmrs.wmnet,service=ats-be |
[production] |
| 15:50 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp4052.drmrs.wmnet,service=ats-tls |
[production] |
| 15:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2169:3317 (T318955)', diff saved to https://phabricator.wikimedia.org/P37215 and previous config saved to /var/cache/conftool/dbconfig/20221031-154638-ladsgroup.json |
[production] |
| 15:46 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance |
[production] |
| 15:46 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance |
[production] |
| 15:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T318955)', diff saved to https://phabricator.wikimedia.org/P37214 and previous config saved to /var/cache/conftool/dbconfig/20221031-154627-ladsgroup.json |
[production] |
| 15:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P37213 and previous config saved to /var/cache/conftool/dbconfig/20221031-154413-ladsgroup.json |
[production] |
| 15:43 |
<jdrewniak@deploy1002> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:851092| Bumping portals to master (T128546)]] (duration: 03m 34s) |
[production] |
| 15:39 |
<jdrewniak@deploy1002> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:851092| Bumping portals to master (T128546)]] (duration: 03m 43s) |
[production] |
| 15:39 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 15:38 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |