301-350 of 10000 results (71ms)
2023-05-16 §
05:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1221 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48239 and previous config saved to /var/cache/conftool/dbconfig/20230516-052936-root.json [production]
05:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48238 and previous config saved to /var/cache/conftool/dbconfig/20230516-052920-root.json [production]
05:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1221 T336337', diff saved to https://phabricator.wikimedia.org/P48237 and previous config saved to /var/cache/conftool/dbconfig/20230516-052026-root.json [production]
05:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121 T336337', diff saved to https://phabricator.wikimedia.org/P48236 and previous config saved to /var/cache/conftool/dbconfig/20230516-052014-root.json [production]
03:54 <mwpresync@deploy1002> Pruned MediaWiki: 1.41.0-wmf.6, 1.41.0-wmf.7 (duration: 02m 26s) [production]
03:51 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.41.0-wmf.9 refs T330215 (duration: 48m 47s) [production]
03:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.41.0-wmf.9 refs T330215 [production]
2023-05-15 §
23:37 <eileen> civicrm upgraded from db6e8d69 to ef7b3822 [production]
22:02 <maryum> deployed patch for T323651 [production]
21:51 <maryum> Deployed patch for T335612 [production]
21:42 <ejegg> payments-wiki upgraded from c0da741f to 8988a598 (and globalcollect settings deleted) [production]
20:00 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
20:00 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:56 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:56 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
19:55 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2003.codfw.wmnet [production]
19:54 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1003.eqiad.wmnet [production]
19:50 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic[2050-2054,2060,2067-2068,2072,2084-2086]* for row D switch upgrade - bking@cumin1001 - T335042 [production]
19:50 <bking@cumin1001> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic[2050-2054,2060,2067-2068,2072,2084-2086]* for row D switch upgrade - bking@cumin1001 - T335042 [production]
19:50 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc-gp2003.codfw.wmnet [production]
19:49 <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic[2050-2054,2060,2067-2068,2072,2084-2086] for row D switch upgrade - bking@cumin1001 - T335042 [production]
19:49 <bking@cumin1001> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic[2050-2054,2060,2067-2068,2072,2084-2086] for row D switch upgrade - bking@cumin1001 - T335042 [production]
19:47 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc-gp1003.eqiad.wmnet [production]
19:47 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 2:00:00 on 20 hosts with reason: T335042 maintenance [production]
19:47 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 2:00:00 on 20 hosts with reason: T335042 maintenance [production]
19:40 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2002.codfw.wmnet [production]
19:37 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1002.eqiad.wmnet [production]
19:33 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc-gp1002.eqiad.wmnet [production]
19:32 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc-gp2002.codfw.wmnet [production]
19:28 <bking@deploy1002> Finished deploy [wdqs/wdqs@41174d5] (wcqs): deploy 0.3.124 to WCQS (duration: 02m 03s) [production]
19:26 <bking@deploy1002> Started deploy [wdqs/wdqs@41174d5] (wcqs): deploy 0.3.124 to WCQS [production]
19:23 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1001.eqiad.wmnet [production]
19:22 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2001.codfw.wmnet [production]
19:19 <bking@deploy1002> Finished deploy [wdqs/wdqs@41174d5]: (no justification provided) (duration: 00m 05s) [production]
19:19 <bking@deploy1002> Started deploy [wdqs/wdqs@41174d5]: (no justification provided) [production]
19:18 <bking@deploy1002> Finished deploy [wdqs/wdqs@41174d5]: (no justification provided) (duration: 00m 05s) [production]
19:18 <bking@deploy1002> Started deploy [wdqs/wdqs@41174d5]: (no justification provided) [production]
19:18 <bking@deploy1002> Finished deploy [wdqs/wdqs@41174d5]: (no justification provided) (duration: 05m 46s) [production]
19:15 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc-gp1001.eqiad.wmnet [production]
19:15 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc-gp2001.codfw.wmnet [production]
19:12 <bking@deploy1002> Started deploy [wdqs/wdqs@41174d5]: (no justification provided) [production]
19:12 <bking@deploy1002> Finished deploy [wdqs/wdqs@41174d5]: 0.3.124 (duration: 10m 05s) [production]
19:03 <inflatador> [WDQS Deploy] Tests passing following deploy of `0.3.124` on canary `wdqs1003`; proceeding to rest of fleet [production]
19:02 <bking@deploy1002> Started deploy [wdqs/wdqs@41174d5]: 0.3.124 [production]
18:54 <mutante> LDAP - added uid 'adee' to groups wmde and nda - T336434 [production]
18:54 <sukhe> set routing-options static route 208.80.153.231/32 next-hop [ 208.80.153.48 208.80.153.10 ]: codfw row D maint 2023/05/16 [dns2002] T335042 [production]
18:33 <brett> Rolling out maglev LVS scheduler in eqsin - T263797 [production]
18:11 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns2005.wikimedia.org with OS bullseye [production]
18:11 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host dns2005.wikimedia.org with OS bullseye [production]
18:06 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns2005.wikimedia.org with OS bullseye [production]