7651-7700 of 10000 results (78ms)
2022-05-27 §
00:45 <mutante> rsyncing /srv/gitlab-backup from gitlab1004 to gitlab2002 | systemctl status full-backup ..in progress on gitlab1001 - T274463 [production]
00:26 <mutante> gitlab1001 deleted backups from last 3 days after rsync to gitlab1004 - freeing disk space, starting the full-backup service once again, should finish now without running out of disk - T274463 [production]
00:26 <mutante> gitlab1001 deleted backups from last 3 days after rsync to gitlab1004 - freeing disk space, starting the full-backup service once again, should finish now without running out of disk - T2744463 [production]
00:05 <mutante> rsyncing /root and /mnt/gitlab-backup of gitlab1001 to /srv/gitlab-backup on gitlab1004 (/srv/gitlab-backup was automounted after creating it and has > 200G free) T274463 [production]
2022-05-26 §
22:22 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:20 <mutante> phabricator adding mabualruz to WMF-NDA group for accest to private tickets T309215 [production]
22:18 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
22:12 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:53 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
21:42 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:27 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1026.mgmt.eqiad.wmnet with reboot policy FORCED [production]
21:24 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
21:17 <mutante> gitlab1004/puppetmaster1001 - create new signing request, sign new cert for puppet, fixed puppet run - T309259 [production]
21:16 <mutante> gitlab1004 - rm -rf /var/lib/puppet/ssl (T309259) [production]
21:15 <mutante> puppetmaster1001 - sudo puppet cert clean gitlab1004.wikimedia.org revoked cert with serial 9600 AND cert with serial 9694 - somehow agent got "cert revoked" before I did anything (T309259) [production]
21:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132 (T298555)', diff saved to https://phabricator.wikimedia.org/P28617 and previous config saved to /var/cache/conftool/dbconfig/20220526-211140-ladsgroup.json [production]
21:10 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1004.wikimedia.org with OS bullseye [production]
20:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P28616 and previous config saved to /var/cache/conftool/dbconfig/20220526-205635-ladsgroup.json [production]
20:55 <brennen> end of utc late backport and config window [production]
20:55 <brennen@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:800190|Revert "Revert "Start writing to cuc_actor in s3, kcgwiki and labtestwiki"" (T233004)]] (duration: 00m 50s) [production]
20:53 <brennen@deploy1002> Synchronized php-1.39.0-wmf.13/extensions/CheckUser/src/Hooks.php: Backport: [[gerrit:798818|Acquire fresh actor id (T233004 T309148)]] (duration: 00m 51s) [production]
20:53 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:52 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:52 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:51 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:47 <ejegg> updated payments-wiki from 592c6d34 to c7c2df36 [production]
20:42 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host cloudelastic1004.wikimedia.org with OS bullseye [production]
20:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P28615 and previous config saved to /var/cache/conftool/dbconfig/20220526-204130-ladsgroup.json [production]
20:40 <inflatador> bking@install1003 removed cloudelastic1004.conf pxe config file T309343 [production]
20:30 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:30 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1004.wikimedia.org with OS bullseye [production]
20:30 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host cloudelastic1004.wikimedia.org with OS bullseye [production]
20:30 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:28 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:28 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1004.wikimedia.org with OS bullseye [production]
20:27 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host cloudelastic1004.wikimedia.org with OS bullseye [production]
20:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132 (T298555)', diff saved to https://phabricator.wikimedia.org/P28614 and previous config saved to /var/cache/conftool/dbconfig/20220526-202625-ladsgroup.json [production]
20:26 <bking@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudelastic1004.wikimedia.org with OS bullseye [production]
20:24 <brennen@deploy1002> Synchronized php-1.39.0-wmf.13/extensions/CheckUser/src/Specials/SpecialCheckUser.php: Backport: [[gerrit:798819|Fix phan failure PhanPluginSimplifyExpressionBool]] (duration: 00m 52s) [production]
20:21 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host cloudelastic1004.wikimedia.org with OS bullseye [production]
20:18 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:18 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:16 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:12 <brennen@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:800247|Enable GrowthExperiments link recommendations, round 4 (T304548)]] (duration: 00m 56s) [production]
19:46 <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (2 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster reimage - bking@cumin1001 - T309343 [production]
19:40 <tgr> T304548 running extensions/GrowthExperiments/maintenance/changeWikiConfig.php on tier4 Growth wikis [production]
19:36 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1004.wikimedia.org with OS bullseye [production]
19:33 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1179 (T298560)', diff saved to https://phabricator.wikimedia.org/P28613 and previous config saved to /var/cache/conftool/dbconfig/20220526-193347-ladsgroup.json [production]