251-300 of 10000 results (28ms)
2021-09-30 ยง
16:08 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:725019|Enable dispatching via job to 10 prod wikis]] (duration: 01m 09s) [production]
15:55 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . [production]
14:36 <elukey> drop /etc/helmfile-defaults/private/backup_old_paths from deploy1002 (old data not needed anymore) [production]
14:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1111 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17374 and previous config saved to /var/cache/conftool/dbconfig/20210930-143325-root.json [production]
14:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17373 and previous config saved to /var/cache/conftool/dbconfig/20210930-143044-root.json [production]
14:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1111 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17372 and previous config saved to /var/cache/conftool/dbconfig/20210930-141822-root.json [production]
14:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17370 and previous config saved to /var/cache/conftool/dbconfig/20210930-141540-root.json [production]
14:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1111 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17369 and previous config saved to /var/cache/conftool/dbconfig/20210930-140318-root.json [production]
14:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17368 and previous config saved to /var/cache/conftool/dbconfig/20210930-140037-root.json [production]
13:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1111 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17367 and previous config saved to /var/cache/conftool/dbconfig/20210930-134815-root.json [production]
13:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17366 and previous config saved to /var/cache/conftool/dbconfig/20210930-134533-root.json [production]
13:42 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet [production]
13:40 <jgiannelos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
13:38 <akosiaris@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
13:38 <akosiaris@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
13:37 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
13:36 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
13:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1111 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17365 and previous config saved to /var/cache/conftool/dbconfig/20210930-133311-root.json [production]
13:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17364 and previous config saved to /var/cache/conftool/dbconfig/20210930-133029-root.json [production]
13:29 <marostegui> Upgrade db1111 [production]
13:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1111 for upgrade', diff saved to https://phabricator.wikimedia.org/P17363 and previous config saved to /var/cache/conftool/dbconfig/20210930-132831-marostegui.json [production]
13:27 <marostegui> Upgrade db1134 [production]
13:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1134 (s1) for upgrade', diff saved to https://phabricator.wikimedia.org/P17362 and previous config saved to /var/cache/conftool/dbconfig/20210930-132700-marostegui.json [production]
13:26 <marostegui> Upgrade db1133 [production]
13:26 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet [production]
13:02 <urbanecm> Start server-side upload for 2 video files (T292096, T291492) [production]
13:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17361 and previous config saved to /var/cache/conftool/dbconfig/20210930-130116-root.json [production]
13:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17360 and previous config saved to /var/cache/conftool/dbconfig/20210930-130109-root.json [production]
12:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17359 and previous config saved to /var/cache/conftool/dbconfig/20210930-124612-root.json [production]
12:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17358 and previous config saved to /var/cache/conftool/dbconfig/20210930-124606-root.json [production]
12:31 <Reedy> downloading files for T290900 in screen on mwmaint1002 [production]
12:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17357 and previous config saved to /var/cache/conftool/dbconfig/20210930-123109-root.json [production]
12:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17356 and previous config saved to /var/cache/conftool/dbconfig/20210930-123101-root.json [production]
12:18 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors (duration: 00m 17s) [production]
12:18 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors [production]
12:17 <moritzm> adapted MX records to point to both mx1001.wikimedia.org and mx2001.wikimedia.org with equal weights T286911 [production]
12:17 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors (duration: 00m 16s) [production]
12:16 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors [production]
12:16 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17355 and previous config saved to /var/cache/conftool/dbconfig/20210930-121605-root.json [production]
12:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17354 and previous config saved to /var/cache/conftool/dbconfig/20210930-121558-root.json [production]
12:14 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors (duration: 00m 15s) [production]
12:13 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors [production]
12:13 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors (duration: 00m 15s) [production]
12:13 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors [production]
12:11 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors (duration: 00m 10s) [production]
12:10 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors [production]
12:10 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors (duration: 00m 01s) [production]
12:10 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@35b9174]: tegola: remove mirror_threshold variable because of parsing errors [production]
12:03 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17353 and previous config saved to /var/cache/conftool/dbconfig/20210930-120102-root.json [production]