|
2022-03-28
ยง
|
| 12:32 |
<jayme@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop: apply |
[production] |
| 12:31 |
<jayme@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop: apply |
[production] |
| 12:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1148 (T298556)', diff saved to https://phabricator.wikimedia.org/P23402 and previous config saved to /var/cache/conftool/dbconfig/20220328-123015-marostegui.json |
[production] |
| 12:30 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1148.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298556)', diff saved to https://phabricator.wikimedia.org/P23401 and previous config saved to /var/cache/conftool/dbconfig/20220328-123007-marostegui.json |
[production] |
| 12:16 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2031.codfw.wmnet with reason: host reimage |
[production] |
| 12:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P23400 and previous config saved to /var/cache/conftool/dbconfig/20220328-121501-marostegui.json |
[production] |
| 12:13 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2031.codfw.wmnet with reason: host reimage |
[production] |
| 11:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P23399 and previous config saved to /var/cache/conftool/dbconfig/20220328-115956-marostegui.json |
[production] |
| 11:55 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp2031.codfw.wmnet with OS buster |
[production] |
| 11:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298556)', diff saved to https://phabricator.wikimedia.org/P23398 and previous config saved to /var/cache/conftool/dbconfig/20220328-114451-marostegui.json |
[production] |
| 11:44 |
<mmandere> |
depool cp2031 for reimage - T290005 |
[production] |
| 11:40 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet |
[production] |
| 11:36 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet |
[production] |
| 11:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet |
[production] |
| 11:30 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet |
[production] |
| 11:25 |
<moritzm> |
installing Intel microcode updates 2022-02-07 on Bullseye |
[production] |
| 11:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1147 (T298556)', diff saved to https://phabricator.wikimedia.org/P23397 and previous config saved to /var/cache/conftool/dbconfig/20220328-112352-marostegui.json |
[production] |
| 11:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1147.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1147.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298556)', diff saved to https://phabricator.wikimedia.org/P23396 and previous config saved to /var/cache/conftool/dbconfig/20220328-112345-marostegui.json |
[production] |
| 11:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P23395 and previous config saved to /var/cache/conftool/dbconfig/20220328-110839-marostegui.json |
[production] |
| 10:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P23394 and previous config saved to /var/cache/conftool/dbconfig/20220328-105333-marostegui.json |
[production] |
| 10:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298556)', diff saved to https://phabricator.wikimedia.org/P23393 and previous config saved to /var/cache/conftool/dbconfig/20220328-103828-marostegui.json |
[production] |
| 10:34 |
<hashar> |
contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` T220606 |
[releng] |
| 10:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1169 (T300775)', diff saved to https://phabricator.wikimedia.org/P23392 and previous config saved to /var/cache/conftool/dbconfig/20220328-102915-marostegui.json |
[production] |
| 10:29 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:29 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:25 |
<hashar> |
Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted |
[releng] |
| 10:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P23391 and previous config saved to /var/cache/conftool/dbconfig/20220328-102014-root.json |
[production] |
| 10:18 |
<hashar> |
contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`. They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943 |
[releng] |
| 10:17 |
<mmandere> |
pool cp2033 with HAProxy as TLS termination layer - T290005 |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1144:3314 (T298556)', diff saved to https://phabricator.wikimedia.org/P23390 and previous config saved to /var/cache/conftool/dbconfig/20220328-101712-marostegui.json |
[production] |
| 10:17 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:17 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298556)', diff saved to https://phabricator.wikimedia.org/P23389 and previous config saved to /var/cache/conftool/dbconfig/20220328-101704-marostegui.json |
[production] |
| 10:13 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2033.codfw.wmnet with OS buster |
[production] |
| 10:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P23387 and previous config saved to /var/cache/conftool/dbconfig/20220328-100511-root.json |
[production] |
| 10:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P23386 and previous config saved to /var/cache/conftool/dbconfig/20220328-100159-marostegui.json |
[production] |
| 09:53 |
<hashar> |
Tag Quibble 1.4.5 @ abe16d574 | T291549 |
[releng] |
| 09:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P23385 and previous config saved to /var/cache/conftool/dbconfig/20220328-095007-root.json |
[production] |
| 09:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P23384 and previous config saved to /var/cache/conftool/dbconfig/20220328-094653-marostegui.json |
[production] |
| 09:46 |
<moritzm> |
installing Linux 4.9.303 on Stretch hosts |
[production] |
| 09:45 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2033.codfw.wmnet with reason: host reimage |
[production] |
| 09:43 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2033.codfw.wmnet with reason: host reimage |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P23383 and previous config saved to /var/cache/conftool/dbconfig/20220328-093503-root.json |
[production] |
| 09:32 |
<wm-bot> |
cleaned up grid queue errors on tools-sgegrid-master.tools.eqiad1.wikimedia.cloud (T304816) - cookbook ran by arturo@nostromo |
[tools] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298556)', diff saved to https://phabricator.wikimedia.org/P23382 and previous config saved to /var/cache/conftool/dbconfig/20220328-093148-marostegui.json |
[production] |
| 09:24 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp2033.codfw.wmnet with OS buster |
[production] |