1501-1550 of 10000 results (33ms)
2022-03-07 ยง
14:46 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance [production]
14:46 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance [production]
14:46 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance [production]
14:46 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance [production]
14:45 <vgutierrez> pool cp1085 with HAProxy as TLS termination layer - T290005 [production]
14:42 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host build2001.codfw.wmnet [production]
14:37 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1002.eqiad.wmnet [production]
14:37 <urbanecm@deploy1002> Synchronized static/images/project-logos/: f50c4746c5fa733929b80b036eef4eee84cf17d1: Revert "Change temporary logo for slwiki" (T302661; 2/2) (duration: 00m 48s) [production]
14:36 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
14:36 <urbanecm@deploy1002> Synchronized wmf-config/logos.php: f50c4746c5fa733929b80b036eef4eee84cf17d1: Revert "Change temporary logo for slwiki" (T302661; 1/2) (duration: 00m 49s) [production]
14:35 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
14:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
14:35 <ntsako@deploy1002> Finished deploy [airflow-dags/analytics@46d88a2]: (no justification provided) (duration: 00m 04s) [production]
14:35 <ntsako@deploy1002> Started deploy [airflow-dags/analytics@46d88a2]: (no justification provided) [production]
14:35 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host failoid1002.eqiad.wmnet [production]
14:34 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4030.ulsfo.wmnet with reason: host reimage [production]
14:34 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
14:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 25%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P21994 and previous config saved to /var/cache/conftool/dbconfig/20220307-143419-root.json [production]
14:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad1003.eqiad.wmnet [production]
14:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142 (T302950)', diff saved to https://phabricator.wikimedia.org/P21993 and previous config saved to /var/cache/conftool/dbconfig/20220307-143229-ladsgroup.json [production]
14:31 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4030.ulsfo.wmnet with reason: host reimage [production]
14:30 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host etherpad1003.eqiad.wmnet [production]
14:30 <moritzm> rebooting etherpad1003 (running etherpad1003) for kernel update [production]
14:29 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2002.codfw.wmnet [production]
14:28 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage [production]
14:28 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1004.wikimedia.org [production]
14:27 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host failoid2002.codfw.wmnet [production]
14:26 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ldap-replica1004.wikimedia.org [production]
14:25 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage [production]
14:23 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1002.eqiad.wmnet [production]
14:22 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1003.wikimedia.org [production]
14:21 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host puppetboard1002.eqiad.wmnet [production]
14:20 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ldap-replica1003.wikimedia.org [production]
14:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 10%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P21992 and previous config saved to /var/cache/conftool/dbconfig/20220307-141915-root.json [production]
14:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 100%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P21991 and previous config saved to /var/cache/conftool/dbconfig/20220307-141911-root.json [production]
14:19 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
14:18 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 64b128459f04514cd0093745d7a83166555449b2: Enable reply tool by default on enwiki (T296645) (duration: 00m 49s) [production]
14:18 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
14:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
14:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P21990 and previous config saved to /var/cache/conftool/dbconfig/20220307-141724-ladsgroup.json [production]
14:16 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
14:16 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 8f20ec9f5bd7f507580d8e8860116e3b1842ac9a: fawiki: Disable creating community books and remove "Create a book" link from sidebar (T303173) (duration: 00m 49s) [production]
14:15 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reimage for host cp4030.ulsfo.wmnet with OS buster [production]
14:15 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2006.wikimedia.org [production]
14:13 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ldap-replica2006.wikimedia.org [production]
14:11 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
14:10 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
14:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
14:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
14:09 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2005.wikimedia.org [production]