|
2022-05-27
§
|
| 08:58 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:07 |
<_joe_> |
restarted rsyslog on kubernetes1014 |
[production] |
| 06:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 100%: After ugprading mysql', diff saved to https://phabricator.wikimedia.org/P28627 and previous config saved to /var/cache/conftool/dbconfig/20220527-065029-root.json |
[production] |
| 06:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 75%: After ugprading mysql', diff saved to https://phabricator.wikimedia.org/P28626 and previous config saved to /var/cache/conftool/dbconfig/20220527-063525-root.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 50%: After ugprading mysql', diff saved to https://phabricator.wikimedia.org/P28625 and previous config saved to /var/cache/conftool/dbconfig/20220527-062022-root.json |
[production] |
| 06:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 25%: After ugprading mysql', diff saved to https://phabricator.wikimedia.org/P28624 and previous config saved to /var/cache/conftool/dbconfig/20220527-060518-root.json |
[production] |
| 05:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 10%: After ugprading mysql', diff saved to https://phabricator.wikimedia.org/P28623 and previous config saved to /var/cache/conftool/dbconfig/20220527-055014-root.json |
[production] |
| 05:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 5%: After ugprading mysql', diff saved to https://phabricator.wikimedia.org/P28622 and previous config saved to /var/cache/conftool/dbconfig/20220527-053510-root.json |
[production] |
| 02:46 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance |
[production] |
| 02:46 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 6 hosts with reason: Maintenance |
[production] |
| 02:46 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance |
[production] |
| 02:46 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance |
[production] |
| 02:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298560)', diff saved to https://phabricator.wikimedia.org/P28621 and previous config saved to /var/cache/conftool/dbconfig/20220527-024627-ladsgroup.json |
[production] |
| 02:40 |
<ejegg> |
updated payments-wiki from 7b65d520 to 59c7239e |
[production] |
| 02:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P28620 and previous config saved to /var/cache/conftool/dbconfig/20220527-023122-ladsgroup.json |
[production] |
| 02:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P28619 and previous config saved to /var/cache/conftool/dbconfig/20220527-021616-ladsgroup.json |
[production] |
| 02:07 |
<ejegg> |
updated payments-wiki from c7c2df36 to 7b65d520 |
[production] |
| 02:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298560)', diff saved to https://phabricator.wikimedia.org/P28618 and previous config saved to /var/cache/conftool/dbconfig/20220527-020111-ladsgroup.json |
[production] |
| 00:45 |
<mutante> |
rsyncing /srv/gitlab-backup from gitlab1004 to gitlab2002 | systemctl status full-backup ..in progress on gitlab1001 - T274463 |
[production] |
| 00:26 |
<mutante> |
gitlab1001 deleted backups from last 3 days after rsync to gitlab1004 - freeing disk space, starting the full-backup service once again, should finish now without running out of disk - T274463 |
[production] |
| 00:26 |
<mutante> |
gitlab1001 deleted backups from last 3 days after rsync to gitlab1004 - freeing disk space, starting the full-backup service once again, should finish now without running out of disk - T2744463 |
[production] |
| 00:05 |
<mutante> |
rsyncing /root and /mnt/gitlab-backup of gitlab1001 to /srv/gitlab-backup on gitlab1004 (/srv/gitlab-backup was automounted after creating it and has > 200G free) T274463 |
[production] |
|
2022-05-26
§
|
| 22:22 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:20 |
<mutante> |
phabricator adding mabualruz to WMF-NDA group for accest to private tickets T309215 |
[production] |
| 22:18 |
<cmooney@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:12 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 21:53 |
<cmooney@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:42 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 21:27 |
<volans@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1026.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 21:24 |
<cmooney@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:17 |
<mutante> |
gitlab1004/puppetmaster1001 - create new signing request, sign new cert for puppet, fixed puppet run - T309259 |
[production] |
| 21:16 |
<mutante> |
gitlab1004 - rm -rf /var/lib/puppet/ssl (T309259) |
[production] |
| 21:15 |
<mutante> |
puppetmaster1001 - sudo puppet cert clean gitlab1004.wikimedia.org revoked cert with serial 9600 AND cert with serial 9694 - somehow agent got "cert revoked" before I did anything (T309259) |
[production] |
| 21:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1132 (T298555)', diff saved to https://phabricator.wikimedia.org/P28617 and previous config saved to /var/cache/conftool/dbconfig/20220526-211140-ladsgroup.json |
[production] |
| 21:10 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1004.wikimedia.org with OS bullseye |
[production] |
| 20:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P28616 and previous config saved to /var/cache/conftool/dbconfig/20220526-205635-ladsgroup.json |
[production] |
| 20:55 |
<brennen> |
end of utc late backport and config window |
[production] |
| 20:55 |
<brennen@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:800190|Revert "Revert "Start writing to cuc_actor in s3, kcgwiki and labtestwiki"" (T233004)]] (duration: 00m 50s) |
[production] |
| 20:53 |
<brennen@deploy1002> |
Synchronized php-1.39.0-wmf.13/extensions/CheckUser/src/Hooks.php: Backport: [[gerrit:798818|Acquire fresh actor id (T233004 T309148)]] (duration: 00m 51s) |
[production] |
| 20:53 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:52 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:52 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:51 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:47 |
<ejegg> |
updated payments-wiki from 592c6d34 to c7c2df36 |
[production] |
| 20:42 |
<bking@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudelastic1004.wikimedia.org with OS bullseye |
[production] |
| 20:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P28615 and previous config saved to /var/cache/conftool/dbconfig/20220526-204130-ladsgroup.json |
[production] |
| 20:40 |
<inflatador> |
bking@install1003 removed cloudelastic1004.conf pxe config file T309343 |
[production] |
| 20:30 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:30 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1004.wikimedia.org with OS bullseye |
[production] |
| 20:30 |
<bking@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudelastic1004.wikimedia.org with OS bullseye |
[production] |