2351-2400 of 10000 results (68ms)
2023-05-30 §
03:54 <mwpresync@deploy1002> Pruned MediaWiki: 1.41.0-wmf.9 (duration: 02m 10s) [production]
03:52 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.41.0-wmf.11 refs T337525 (duration: 49m 54s) [production]
03:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.41.0-wmf.11 refs T337525 [production]
2023-05-29 §
15:19 <eoghan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: This is being worked on [production]
15:19 <eoghan@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: This is being worked on [production]
14:18 <stevemunene@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on stat1009.eqiad.wmnet with reason: Bringing stat1009 into service [production]
14:18 <stevemunene@cumin1001> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on stat1009.eqiad.wmnet with reason: Bringing stat1009 into service [production]
13:57 <vgutierrez@puppetmaster1001> conftool action : set/weight=10; selector: name=dbproxy.*,dc=eqiad [production]
11:25 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
11:24 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
11:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1196 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48618 and previous config saved to /var/cache/conftool/dbconfig/20230529-112242-root.json [production]
11:13 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
11:13 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
11:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1196 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48617 and previous config saved to /var/cache/conftool/dbconfig/20230529-110737-root.json [production]
10:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1196 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48616 and previous config saved to /var/cache/conftool/dbconfig/20230529-105233-root.json [production]
10:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1196 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48615 and previous config saved to /var/cache/conftool/dbconfig/20230529-103728-root.json [production]
10:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1196 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48614 and previous config saved to /var/cache/conftool/dbconfig/20230529-102223-root.json [production]
10:07 <vgutierrez> restarting pybal on lvs1018 [production]
10:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1196 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48612 and previous config saved to /var/cache/conftool/dbconfig/20230529-100719-root.json [production]
10:05 <oblivian@deploy1002> helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
10:05 <oblivian@deploy1002> helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
10:05 <oblivian@deploy1002> helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
10:05 <oblivian@deploy1002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
10:04 <oblivian@deploy1002> helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
10:04 <oblivian@deploy1002> helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
10:03 <oblivian@deploy1002> helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync [production]
10:03 <oblivian@deploy1002> helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
10:03 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
10:02 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
10:00 <vgutierrez> restarting pybal on lvs1020 [production]
09:59 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
09:58 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
09:56 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
09:55 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
09:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1196 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48611 and previous config saved to /var/cache/conftool/dbconfig/20230529-095214-root.json [production]
09:52 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
09:51 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
09:50 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
09:49 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
09:45 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
09:45 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
09:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1196 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48610 and previous config saved to /var/cache/conftool/dbconfig/20230529-093709-root.json [production]
09:31 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
09:31 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
09:30 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
09:29 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
09:13 <godog> start partial rollout of cadvisor to eqiad/codfw (~10%) T108027 [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1161 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48609 and previous config saved to /var/cache/conftool/dbconfig/20230529-090216-root.json [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1161 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48608 and previous config saved to /var/cache/conftool/dbconfig/20230529-084711-root.json [production]
08:45 <godog> delete old raw blocks from thanos - T337236 [production]