151-200 of 10000 results (39ms)
2022-03-22 ยง
13:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:27 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudmetrics1003.eqiad.wmnet [production]
13:27 <jnuche@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.3 refs T300203 (duration: 00m 52s) [production]
13:26 <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.3 refs T300203 [production]
13:26 <aborrero@cumin2002> START - Cookbook sre.hosts.reboot-single for host cloudgw2001-dev.codfw.wmnet [production]
13:25 <aborrero@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2002-dev.codfw.wmnet [production]
13:20 <aborrero@cumin2002> START - Cookbook sre.hosts.reboot-single for host cloudgw2002-dev.codfw.wmnet [production]
13:19 <aborrero@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudgw2002-dev.codfw.wmnet [production]
13:19 <aborrero@cumin2002> START - Cookbook sre.hosts.reboot-single for host cloudgw2002-dev.codfw.wmnet [production]
12:54 <moritzm> installing 5.10.103 kernels on servers running a kernel from buster backports T303179 [production]
12:52 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
12:52 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
12:51 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance [production]
12:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance [production]
12:51 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
12:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
12:41 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
12:41 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
12:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1100 (re)pooling @ 100%: After reboot', diff saved to https://phabricator.wikimedia.org/P22959 and previous config saved to /var/cache/conftool/dbconfig/20220322-124117-root.json [production]
12:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 100%: After reboot', diff saved to https://phabricator.wikimedia.org/P22958 and previous config saved to /var/cache/conftool/dbconfig/20220322-124109-root.json [production]
12:36 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance [production]
12:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance [production]
12:33 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
12:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
12:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
12:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1132 after testing', diff saved to https://phabricator.wikimedia.org/P22957 and previous config saved to /var/cache/conftool/dbconfig/20220322-123056-marostegui.json [production]
12:28 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
12:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1100 (re)pooling @ 75%: After reboot', diff saved to https://phabricator.wikimedia.org/P22956 and previous config saved to /var/cache/conftool/dbconfig/20220322-122613-root.json [production]
12:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 75%: After reboot', diff saved to https://phabricator.wikimedia.org/P22955 and previous config saved to /var/cache/conftool/dbconfig/20220322-122605-root.json [production]
12:24 <marostegui> dbmaint s3@eqiad T300600 [production]
12:24 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:772817|Enable WRITE BOTH on rest of s6 for templatelinks normalization (T299421)]] (duration: 00m 54s) [production]
12:23 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
12:21 <marostegui> dbmaint s7@eqiad T300992 [production]
12:19 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
12:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
12:18 <marostegui> dbmaint s6@eqiad T300992 [production]
12:17 <marostegui> dbmaint s5@eqiad T300992 [production]
12:16 <marostegui> dbmaint s8@eqiad T300992 [production]
12:15 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
12:12 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:772816|Enable WRITE BOTH for templatelinks normalization in wikitech (T299421)]] (duration: 01m 41s) [production]
12:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1100 (re)pooling @ 50%: After reboot', diff saved to https://phabricator.wikimedia.org/P22954 and previous config saved to /var/cache/conftool/dbconfig/20220322-121110-root.json [production]
12:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 50%: After reboot', diff saved to https://phabricator.wikimedia.org/P22953 and previous config saved to /var/cache/conftool/dbconfig/20220322-121101-root.json [production]
12:01 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
12:01 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
12:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298557)', diff saved to https://phabricator.wikimedia.org/P22952 and previous config saved to /var/cache/conftool/dbconfig/20220322-120123-marostegui.json [production]
11:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db1100 (re)pooling @ 25%: After reboot', diff saved to https://phabricator.wikimedia.org/P22951 and previous config saved to /var/cache/conftool/dbconfig/20220322-115606-root.json [production]
11:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 25%: After reboot', diff saved to https://phabricator.wikimedia.org/P22950 and previous config saved to /var/cache/conftool/dbconfig/20220322-115557-root.json [production]
11:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P22949 and previous config saved to /var/cache/conftool/dbconfig/20220322-114618-marostegui.json [production]
11:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1100 (re)pooling @ 10%: After reboot', diff saved to https://phabricator.wikimedia.org/P22948 and previous config saved to /var/cache/conftool/dbconfig/20220322-114102-root.json [production]