601-650 of 10000 results (42ms)
2022-01-24 ยง
12:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:08 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1025.eqiad.wmnet with OS buster [production]
12:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2026.codfw.wmnet with OS bullseye [production]
12:06 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti1014.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage [production]
12:06 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti1014.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage [production]
12:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1126 (T285149)', diff saved to https://phabricator.wikimedia.org/P19041 and previous config saved to /var/cache/conftool/dbconfig/20220124-120635-marostegui.json [production]
12:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1126.eqiad.wmnet with reason: Maintenance [production]
12:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1126.eqiad.wmnet with reason: Maintenance [production]
12:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T285149)', diff saved to https://phabricator.wikimedia.org/P19040 and previous config saved to /var/cache/conftool/dbconfig/20220124-120627-marostegui.json [production]
12:06 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:06 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 60%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19039 and previous config saved to /var/cache/conftool/dbconfig/20220124-120602-root.json [production]
12:05 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 1a463610ba2a92f7437c6921a9591616de0d242e: fawiki: Remove move-rootuserpages flag from users (T299847) (duration: 00m 49s) [production]
12:00 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1025.eqiad.wmnet with OS buster [production]
11:59 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1024.eqiad.wmnet with OS buster [production]
11:58 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. [production]
11:56 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
11:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove contributions from s8 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P19038 and previous config saved to /var/cache/conftool/dbconfig/20220124-115334-marostegui.json [production]
11:53 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove special groups from s8 codfw T263127', diff saved to https://phabricator.wikimedia.org/P19037 and previous config saved to /var/cache/conftool/dbconfig/20220124-115236-marostegui.json [production]
11:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P19036 and previous config saved to /var/cache/conftool/dbconfig/20220124-115123-marostegui.json [production]
11:50 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 50%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19035 and previous config saved to /var/cache/conftool/dbconfig/20220124-115059-root.json [production]
11:50 <vgutierrez> pool cp1088 using envoy as TLS termination layer - T271421 [production]
11:50 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1024.eqiad.wmnet with OS buster [production]
11:49 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1088.eqiad.wmnet with OS buster [production]
11:48 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
11:37 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host es2026.codfw.wmnet with OS bullseye [production]
11:37 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. [production]
11:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P19034 and previous config saved to /var/cache/conftool/dbconfig/20220124-113618-marostegui.json [production]
11:35 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 40%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19033 and previous config saved to /var/cache/conftool/dbconfig/20220124-113555-root.json [production]
11:31 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1024.eqiad.wmnet with OS buster [production]
11:31 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1024.eqiad.wmnet with OS buster [production]
11:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2031.codfw.wmnet with OS bullseye [production]
11:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T285149)', diff saved to https://phabricator.wikimedia.org/P19032 and previous config saved to /var/cache/conftool/dbconfig/20220124-112113-marostegui.json [production]
11:20 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 25%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19031 and previous config saved to /var/cache/conftool/dbconfig/20220124-112051-root.json [production]
11:19 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1024.eqiad.wmnet with OS buster [production]
11:19 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=restbase1023.eqiad.wmnet [production]
11:17 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1023.eqiad.wmnet with OS buster [production]
11:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2033.codfw.wmnet with OS bullseye [production]
11:05 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 20%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19030 and previous config saved to /var/cache/conftool/dbconfig/20220124-110548-root.json [production]
11:04 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1023.eqiad.wmnet with OS buster [production]
11:03 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reimage for host cp1088.eqiad.wmnet with OS buster [production]
11:02 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=restbase1022.eqiad.wmnet [production]
10:59 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1022.eqiad.wmnet with OS buster [production]
10:58 <vgutierrez> depool cp1088 to be reimaged as cache::upload_envoy - T271421 [production]
10:56 <moritzm> installing modsecurity-apache security updates [production]
10:54 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host es2031.codfw.wmnet with OS bullseye [production]
10:51 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es2031.codfw.wmnet with OS bullseye [production]
10:50 <marostegui@cumin1001> dbctl commit (dc=all): 'es1027 (re)pooling @ 10%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19028 and previous config saved to /var/cache/conftool/dbconfig/20220124-105044-root.json [production]
10:49 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1022.eqiad.wmnet with OS buster [production]
10:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1167 (T285149)', diff saved to https://phabricator.wikimedia.org/P19027 and previous config saved to /var/cache/conftool/dbconfig/20220124-103958-marostegui.json [production]