451-500 of 10000 results (98ms)
2024-10-30 ยง
19:40 <swfrench-wmf> all shellbox instances updated to shellbox 2024-10-15-214239 - T375243 [production]
19:39 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply [production]
19:39 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply [production]
19:37 <mutante> gitlab - deleting user "jfk" on main server and both replicas T376936 [production]
19:37 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply [production]
19:36 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/shellbox-video: apply [production]
19:36 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply [production]
19:35 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply [production]
19:35 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply [production]
19:34 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/shellbox-media: apply [production]
19:34 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/shellbox: apply [production]
19:33 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/shellbox: apply [production]
19:27 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2115 (T376905)', diff saved to https://phabricator.wikimedia.org/P70715 and previous config saved to /var/cache/conftool/dbconfig/20241030-192744-ladsgroup.json [production]
19:20 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2115 (T376905)', diff saved to https://phabricator.wikimedia.org/P70714 and previous config saved to /var/cache/conftool/dbconfig/20241030-192011-ladsgroup.json [production]
19:20 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2115.codfw.wmnet with reason: Maintenance [production]
19:19 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2115.codfw.wmnet with reason: Maintenance [production]
19:17 <dduvall@deploy2002> rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.1 refs T375660 [production]
18:40 <dduvall@deploy2002> Finished scap sync-world: Backport for [[gerrit:1084759|Revert "Use array instead of string for class list" (T378531)]] (duration: 19m 04s) [production]
18:39 <inflatador> bking@stat1008,stat1009,stat1010.mgmt racadm jobqueue delete -i $job T376813 [production]
18:36 <fnegri@cumin1002> END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database nrwiki (T375101) [production]
18:35 <dduvall@deploy2002> ammarpad, dduvall: Continuing with sync [production]
18:35 <dduvall> error is still occurring following backport deployment of https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/1084759 (T378531) [production]
18:27 <dduvall> monitoring testwiki error rates for a few minutes to see if the error related to T378531 subsides (current rate is 23 errors in the last 15 minutes) [production]
18:23 <dduvall@deploy2002> ammarpad, dduvall: Backport for [[gerrit:1084759|Revert "Use array instead of string for class list" (T378531)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
18:21 <dduvall@deploy2002> Started scap sync-world: Backport for [[gerrit:1084759|Revert "Use array instead of string for class list" (T378531)]] [production]
18:10 <fnegri@cumin1002> START - Cookbook sre.wikireplicas.add-wiki for database nrwiki (T375101) [production]
17:35 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet,service=s3 [production]
17:35 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet,service=s1 [production]
17:31 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s3 [production]
17:26 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s1 [production]
17:24 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1018.eqiad.wmnet,service=s7 [production]
17:23 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1018.eqiad.wmnet,service=s7 [production]
17:21 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1019.eqiad.wmnet,service=s6 [production]
17:21 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1019.eqiad.wmnet,service=s4 [production]
17:20 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet,service=s6 [production]
17:20 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet,service=s4 [production]
17:19 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1020.eqiad.wmnet,service=s5 [production]
17:18 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1020.eqiad.wmnet,service=s8 [production]
17:11 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1020.eqiad.wmnet,service=s8 [production]
17:11 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1020.eqiad.wmnet,service=s5 [production]
17:03 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply [production]
17:03 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply [production]
17:01 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply [production]
17:00 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [production]
16:59 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply [production]
16:58 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply [production]
16:58 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply [production]
16:57 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-media: apply [production]
16:54 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [production]
16:53 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox: apply [production]