1351-1400 of 10000 results (89ms)
2023-01-23 ยง
14:25 <taavi@deploy1002> Finished scap: Backport for [[gerrit:882422|bnwikiquote: Update logo (T323131)]], [[gerrit:882425|shnwikibooks: Add project logo (T327380)]] (duration: 09m 22s) [production]
14:25 <elukey@deploy1002> helmfile [staging] DONE helmfile.d/services/changeprop: sync [production]
14:25 <elukey@deploy1002> helmfile [staging] START helmfile.d/services/changeprop: sync [production]
14:20 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
14:20 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
14:18 <taavi> mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=zhwiki pageassessments # T326387 [production]
14:17 <taavi@deploy1002> taavi and stang: Backport for [[gerrit:882422|bnwikiquote: Update logo (T323131)]], [[gerrit:882425|shnwikibooks: Add project logo (T327380)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
14:16 <taavi@deploy1002> Started scap: Backport for [[gerrit:882422|bnwikiquote: Update logo (T323131)]], [[gerrit:882425|shnwikibooks: Add project logo (T327380)]] [production]
12:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2107 (T323827)', diff saved to https://phabricator.wikimedia.org/P43246 and previous config saved to /var/cache/conftool/dbconfig/20230123-124532-ladsgroup.json [production]
12:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P43245 and previous config saved to /var/cache/conftool/dbconfig/20230123-123025-ladsgroup.json [production]
12:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P43242 and previous config saved to /var/cache/conftool/dbconfig/20230123-121519-ladsgroup.json [production]
12:08 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
12:08 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
12:06 <marostegui> dbmaint Reboot db2135 (m5 codfw master) [production]
12:06 <marostegui> dbmaint Reboot db2134 (m3 codfw master) [production]
12:05 <Emperor> removing /usr/local/bin/prometheus-puppet-agent-stats from prometheus crontab on snapshot1014 [production]
12:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2107 (T323827)', diff saved to https://phabricator.wikimedia.org/P43241 and previous config saved to /var/cache/conftool/dbconfig/20230123-120012-ladsgroup.json [production]
11:58 <marostegui> dbmaint Reboot db2133 (m2 codfw master) [production]
11:57 <marostegui> dbmaint Reboot db2132 (m1 codfw master) [production]
11:57 <marostegui> Reboot db2132 (m1 codfw master) [production]
11:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2113 (re)pooling @ 100%: Maint done', diff saved to https://phabricator.wikimedia.org/P43239 and previous config saved to /var/cache/conftool/dbconfig/20230123-113506-ladsgroup.json [production]
11:31 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
11:31 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
11:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
11:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
11:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db2114 T327644', diff saved to https://phabricator.wikimedia.org/P43236 and previous config saved to /var/cache/conftool/dbconfig/20230123-112134-ladsgroup.json [production]
11:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2113 (re)pooling @ 75%: Maint done', diff saved to https://phabricator.wikimedia.org/P43235 and previous config saved to /var/cache/conftool/dbconfig/20230123-112001-ladsgroup.json [production]
11:18 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Promote db2129 to s6 primary T327644', diff saved to https://phabricator.wikimedia.org/P43234 and previous config saved to /var/cache/conftool/dbconfig/20230123-111813-ladsgroup.json [production]
11:17 <Amir1> Starting s6 codfw failover from db2114 to db2129 - T327644 [production]
11:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2107 (T323827)', diff saved to https://phabricator.wikimedia.org/P43233 and previous config saved to /var/cache/conftool/dbconfig/20230123-111147-ladsgroup.json [production]
11:11 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
11:11 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
11:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2113 (re)pooling @ 25%: Maint done', diff saved to https://phabricator.wikimedia.org/P43232 and previous config saved to /var/cache/conftool/dbconfig/20230123-110456-ladsgroup.json [production]
10:55 <XioNoX> update management routers ACLs to add new bast hosts [production]
10:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set db2129 with weight 0 T327644', diff saved to https://phabricator.wikimedia.org/P43231 and previous config saved to /var/cache/conftool/dbconfig/20230123-105520-ladsgroup.json [production]
10:54 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s6 T327644 [production]
10:54 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s6 T327644 [production]
10:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2113 (re)pooling @ 10%: Maint done', diff saved to https://phabricator.wikimedia.org/P43230 and previous config saved to /var/cache/conftool/dbconfig/20230123-104951-ladsgroup.json [production]
10:48 <vgutierrez> rolling upgrade to HAProxy 2.4.20 on ulsfo [production]
10:40 <btullis@deploy1002> Finished deploy [analytics/superset/deploy@4ba1cb1]: (no justification provided) (duration: 00m 06s) [production]
10:40 <btullis@deploy1002> Started deploy [analytics/superset/deploy@4ba1cb1]: (no justification provided) [production]
10:40 <btullis@deploy1002> Finished deploy [analytics/superset/deploy@4ba1cb1]: (no justification provided) (duration: 00m 20s) [production]
10:40 <btullis@deploy1002> Started deploy [analytics/superset/deploy@4ba1cb1]: (no justification provided) [production]
10:39 <btullis@deploy1002> Installation of scap version "4.33.1" completed for 1 hosts [production]
10:39 <btullis@deploy1002> Installing scap version "4.33.1" for 1 hosts [production]
10:37 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-tool1010.eqiad.wmnet with OS bullseye [production]
10:18 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-tool1010.eqiad.wmnet with reason: host reimage [production]
10:16 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-tool1010.eqiad.wmnet with reason: host reimage [production]
10:07 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:877244|Remove Flow as default in techconductwiki]] (duration: 07m 51s) [production]
10:03 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host an-tool1010.eqiad.wmnet with OS bullseye [production]