901-950 of 10000 results (85ms)
2024-02-15 §
16:53 <dani@deploy2002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
16:53 <dani@deploy2002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
16:53 <dani@deploy2002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
16:53 <dani@deploy2002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
16:53 <dani@deploy2002> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
16:53 <dani@deploy2002> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
16:52 <dani@deploy2002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
16:51 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "rack1"} and A:aqs and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002 [production]
16:46 <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@162f72f] (sessionstore): Deploying to updated target list — T353550 (duration: 00m 15s) [production]
16:46 <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@162f72f] (sessionstore): Deploying to updated target list — T353550 [production]
16:46 <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@162f72f] (ml-cache): Deploying to updated target list — T353550 (duration: 00m 15s) [production]
16:46 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw2379.codfw.wmnet with reason: BGP issues - uncordoned, needs investigation [production]
16:45 <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@162f72f] (ml-cache): Deploying to updated target list — T353550 [production]
16:45 <hnowlan@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw2379.codfw.wmnet with reason: BGP issues - uncordoned, needs investigation [production]
16:45 <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@162f72f] (cassandra-dev): Deploying to updated target list — T353550 (duration: 00m 15s) [production]
16:45 <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@162f72f] (cassandra-dev): Deploying to updated target list — T353550 [production]
16:43 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2155 (re)pooling @ 75%: T355866 - Post migration repool of db2155', diff saved to https://phabricator.wikimedia.org/P56840 and previous config saved to /var/cache/conftool/dbconfig/20240215-164348-arnaudb.json [production]
16:43 <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@162f72f] (aqs): Deploying to updated target list — T353550 (duration: 00m 37s) [production]
16:43 <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@162f72f] (aqs): Deploying to updated target list — T353550 [production]
16:40 <dani@deploy2002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
16:40 <dani@deploy2002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
16:40 <hnowlan@cumin2002> conftool action : set/pooled=no; selector: name=mw2379.codfw.wmnet [production]
16:40 <dani@deploy2002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
16:40 <dani@deploy2002> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
16:40 <dani@deploy2002> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
16:38 <dancy@deploy2002> Finished scap: Backport for [[gerrit:1002993|Load WikimediaCampaignEvents if CampaignEvents is loaded (T347909)]] (duration: 13m 36s) [production]
16:30 <dancy@deploy2002> mhorsey and dancy: Continuing with sync [production]
16:29 <hnowlan> kubectl cordon mw2379.codfw.wmnet - bgp issues [production]
16:28 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2155 (re)pooling @ 50%: T355866 - Post migration repool of db2155', diff saved to https://phabricator.wikimedia.org/P56839 and previous config saved to /var/cache/conftool/dbconfig/20240215-162843-arnaudb.json [production]
16:26 <dancy@deploy2002> mhorsey and dancy: Backport for [[gerrit:1002993|Load WikimediaCampaignEvents if CampaignEvents is loaded (T347909)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
16:24 <dancy@deploy2002> Started scap: Backport for [[gerrit:1002993|Load WikimediaCampaignEvents if CampaignEvents is loaded (T347909)]] [production]
16:16 <hnowlan@cumin2002> conftool action : set/weight=10:pooled=yes; selector: name=(mw2311.codfw.wmnet|mw2335.codfw.wmnet|mw2379.codfw.wmnet|mw2380.codfw.wmnet|mw2383.codfw.wmnet),cluster=kubernetes,service=kubesvc [production]
16:14 <Daimona> Creating new DB table for the WikimediaCampaignEvents extension in x1.testwiki, x1.test2wiki, x1.officewiki, and x1.wikishared # T347909 [production]
16:13 <cgoubert@cumin2002> conftool action : set/pooled=yes; selector: name=(mw2302|mw2303|mw2304|mw2305|mw2306|mw2307|mw2308|mw2309|mw2426).* [production]
16:13 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2155 (re)pooling @ 25%: T355866 - Post migration repool of db2155', diff saved to https://phabricator.wikimedia.org/P56838 and previous config saved to /var/cache/conftool/dbconfig/20240215-161338-arnaudb.json [production]
16:13 <claime> Repooling mw2302|mw2303|mw2304|mw2305|mw2306|mw2307|mw2308|mw2309|mw2426 - T355866 [production]
16:13 <claime> Uncordoning kubernetes2059.codfw.wmnet kubernetes2028.codfw.wmnet kubernetes2027.codfw.wmnet kubernetes2060.codfw.wmnet kubernetes2008.codfw.wmnet kubernetes2007.codfw.wmnet kubernetes2055.codfw.wmnet mw2301.codfw.wmnet mw2424.codfw.wmnet mw2425.codfw.wmnet mw2427.codfw.wmnet - T355866 [production]
16:13 <hnowlan@deploy2002> helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:12 <hnowlan@deploy2002> helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:12 <hnowlan@deploy2002> helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
16:12 <hnowlan@deploy2002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
16:00 <topranks> commencing move of server uplinks codfw row A6 T355866 [production]
15:57 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 38 hosts with reason: Migrating servers in codfw rack A6 to lsw1-a6-codfw [production]
15:56 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 38 hosts with reason: Migrating servers in codfw rack A6 to lsw1-a6-codfw [production]
15:54 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on es2028.codfw.wmnet with reason: T355866 - Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw [production]
15:54 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on es2028.codfw.wmnet with reason: T355866 - Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw [production]
15:54 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on es2027.codfw.wmnet with reason: T355866 - Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw [production]
15:53 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on es2027.codfw.wmnet with reason: T355866 - Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw [production]
15:49 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a6-codfw.mgmt with reason: prepping for server uplink migration codfw rack a6 [production]
15:49 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a6-codfw.mgmt with reason: prepping for server uplink migration codfw rack a6 [production]