6401-6450 of 10000 results (46ms)
2022-01-24 ยง
12:22 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:22 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: db340cc: 5424d69: Update wgCopyUploadsDomains allowlist (T299579, T299881) (duration: 00m 48s) [production]
12:21 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 75%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19043 and previous config saved to /var/cache/conftool/dbconfig/20220124-122106-root.json [production]
12:21 <moritzm> installing ICU security updates on stretch [production]
12:19 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 2c7b45a3f080757338a877f4024e27dea8cc47c5: fawiki: Exempt draft namespace from robots control by users (T299850) (duration: 05m 39s) [production]
12:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:14 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1026.eqiad.wmnet with OS buster [production]
12:13 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host es1033.eqiad.wmnet with OS bullseye [production]
12:13 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:11 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:10 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=restbase102[45].eqiad.wmnet [production]
12:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1033 T299889', diff saved to https://phabricator.wikimedia.org/P19042 and previous config saved to /var/cache/conftool/dbconfig/20220124-121029-marostegui.json [production]
12:10 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 97d047bc4bf3748fc76f63647d77d26cc545b49f: Remove kea, nod, and sms from wmfGetVariantSettings (T299304; T296286; T298075; T298182) (duration: 00m 49s) [production]
12:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:08 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1025.eqiad.wmnet with OS buster [production]
12:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2026.codfw.wmnet with OS bullseye [production]
12:06 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti1014.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage [production]
12:06 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti1014.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage [production]
12:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1126 (T285149)', diff saved to https://phabricator.wikimedia.org/P19041 and previous config saved to /var/cache/conftool/dbconfig/20220124-120635-marostegui.json [production]
12:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1126.eqiad.wmnet with reason: Maintenance [production]
12:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1126.eqiad.wmnet with reason: Maintenance [production]
12:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T285149)', diff saved to https://phabricator.wikimedia.org/P19040 and previous config saved to /var/cache/conftool/dbconfig/20220124-120627-marostegui.json [production]
12:06 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:06 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 60%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19039 and previous config saved to /var/cache/conftool/dbconfig/20220124-120602-root.json [production]
12:05 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 1a463610ba2a92f7437c6921a9591616de0d242e: fawiki: Remove move-rootuserpages flag from users (T299847) (duration: 00m 49s) [production]
12:00 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1025.eqiad.wmnet with OS buster [production]
11:59 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1024.eqiad.wmnet with OS buster [production]
11:58 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. [production]
11:56 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
11:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove contributions from s8 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P19038 and previous config saved to /var/cache/conftool/dbconfig/20220124-115334-marostegui.json [production]
11:53 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove special groups from s8 codfw T263127', diff saved to https://phabricator.wikimedia.org/P19037 and previous config saved to /var/cache/conftool/dbconfig/20220124-115236-marostegui.json [production]
11:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P19036 and previous config saved to /var/cache/conftool/dbconfig/20220124-115123-marostegui.json [production]
11:50 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 50%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19035 and previous config saved to /var/cache/conftool/dbconfig/20220124-115059-root.json [production]
11:50 <vgutierrez> pool cp1088 using envoy as TLS termination layer - T271421 [production]
11:50 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1024.eqiad.wmnet with OS buster [production]
11:49 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1088.eqiad.wmnet with OS buster [production]
11:48 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
11:37 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host es2026.codfw.wmnet with OS bullseye [production]
11:37 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. [production]
11:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P19034 and previous config saved to /var/cache/conftool/dbconfig/20220124-113618-marostegui.json [production]
11:35 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 40%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19033 and previous config saved to /var/cache/conftool/dbconfig/20220124-113555-root.json [production]
11:31 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1024.eqiad.wmnet with OS buster [production]
11:31 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1024.eqiad.wmnet with OS buster [production]
11:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2031.codfw.wmnet with OS bullseye [production]
11:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T285149)', diff saved to https://phabricator.wikimedia.org/P19032 and previous config saved to /var/cache/conftool/dbconfig/20220124-112113-marostegui.json [production]
11:20 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 25%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19031 and previous config saved to /var/cache/conftool/dbconfig/20220124-112051-root.json [production]
11:19 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1024.eqiad.wmnet with OS buster [production]
11:19 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=restbase1023.eqiad.wmnet [production]