2151-2200 of 10000 results (37ms)
2021-09-07 §
09:26 <Emperor> removing pc1009 from tendril and zarcillo T289120 [production]
09:25 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1009.eqiad.wmnet [production]
09:16 <mvernon@cumin1001> START - Cookbook sre.hosts.decommission for hosts pc1009.eqiad.wmnet [production]
08:57 <elukey@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:53 <elukey@cumin1001> START - Cookbook sre.dns.netbox [production]
08:51 <Emperor> removing pc1008 from orchestrator T289119 [production]
08:44 <Emperor> removing pc1008 from tendril and zarcillo T289119 [production]
08:42 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1008.eqiad.wmnet [production]
08:31 <mvernon@cumin1001> START - Cookbook sre.hosts.decommission for hosts pc1008.eqiad.wmnet [production]
08:29 <marostegui@cumin1001> dbctl commit (dc=all): 'More weight for db2090 into API T288803', diff saved to https://phabricator.wikimedia.org/P17241 and previous config saved to /var/cache/conftool/dbconfig/20210907-082952-marostegui.json [production]
08:25 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
08:25 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
08:25 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
08:24 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
08:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 100%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17240 and previous config saved to /var/cache/conftool/dbconfig/20210907-080230-root.json [production]
07:52 <kormat@cumin1001> dbctl commit (dc=all): 'db2118 (re)pooling @ 100%: reimage to buster (now with fixed pool config) T288244', diff saved to https://phabricator.wikimedia.org/P17239 and previous config saved to /var/cache/conftool/dbconfig/20210907-075235-kormat.json [production]
07:49 <marostegui@cumin1001> dbctl commit (dc=all): 'More weight for db2090 into API T288803', diff saved to https://phabricator.wikimedia.org/P17238 and previous config saved to /var/cache/conftool/dbconfig/20210907-074901-marostegui.json [production]
07:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 75%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17237 and previous config saved to /var/cache/conftool/dbconfig/20210907-074726-root.json [production]
07:37 <kormat@cumin1001> dbctl commit (dc=all): 'db2118 (re)pooling @ 75%: reimage to buster (now with fixed pool config) T288244', diff saved to https://phabricator.wikimedia.org/P17236 and previous config saved to /var/cache/conftool/dbconfig/20210907-073731-kormat.json [production]
07:37 <godog> +100G for prometheus/k8s codfw [production]
07:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Start to pool db2090 into API T288803', diff saved to https://phabricator.wikimedia.org/P17235 and previous config saved to /var/cache/conftool/dbconfig/20210907-073436-marostegui.json [production]
07:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 50%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17234 and previous config saved to /var/cache/conftool/dbconfig/20210907-073222-root.json [production]
07:22 <kormat@cumin1001> dbctl commit (dc=all): 'db2118 (re)pooling @ 50%: reimage to buster (now with fixed pool config) T288244', diff saved to https://phabricator.wikimedia.org/P17233 and previous config saved to /var/cache/conftool/dbconfig/20210907-072227-kormat.json [production]
07:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 25%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17232 and previous config saved to /var/cache/conftool/dbconfig/20210907-071719-root.json [production]
07:13 <jayme@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
07:13 <jayme@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
07:07 <kormat@cumin1001> dbctl commit (dc=all): 'db2118 (re)pooling @ 25%: reimage to buster (now with fixed pool config) T288244', diff saved to https://phabricator.wikimedia.org/P17231 and previous config saved to /var/cache/conftool/dbconfig/20210907-070724-kormat.json [production]
07:07 <kormat@cumin1001> dbctl commit (dc=all): 'Fixing db2118's pooling config T288244', diff saved to https://phabricator.wikimedia.org/P17230 and previous config saved to /var/cache/conftool/dbconfig/20210907-070702-kormat.json [production]
07:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 10%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17229 and previous config saved to /var/cache/conftool/dbconfig/20210907-070215-root.json [production]
06:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 5%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17228 and previous config saved to /var/cache/conftool/dbconfig/20210907-064711-root.json [production]
05:15 <marostegui> Optimize eowiki.flaggedtemplates in eqiad T290057 [production]
05:15 <marostegui> Optimize vecwiki.flaggedtemplates in eqiad T290057 [production]
05:14 <marostegui> Optimize kawiki.flaggedtemplates in eqiad T290057 [production]
2021-09-06 §
23:52 <tstarling@deploy1002> Synchronized php-1.37.0-wmf.21/extensions/SecurePoll/includes/Talliers/STVTallier.php: T290000 (duration: 00m 58s) [production]
16:14 <Amir1> Deployed patch for T290394 [production]
15:01 <Emperor> removing pc1007 from orchestrator T289118 [production]
15:00 <jiji@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:53 <kormat@cumin1001> dbctl commit (dc=all): 'db2118 (re)pooling @ 25%: reimage to buster T288244', diff saved to https://phabricator.wikimedia.org/P17226 and previous config saved to /var/cache/conftool/dbconfig/20210906-145341-kormat.json [production]
14:49 <Emperor> removing pc1007 from tendril and zarcillo T289118 [production]
14:45 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1007.eqiad.wmnet [production]
14:45 <volans@cumin1001> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) for hosts mc1026.eqiad.wmnet [production]
14:44 <volans@cumin1001> START - Cookbook sre.hosts.decommission for hosts mc1026.eqiad.wmnet [production]
14:36 <volans@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mc1027.eqiad.wmnet [production]
14:35 <mvernon@cumin1001> START - Cookbook sre.hosts.decommission for hosts pc1007.eqiad.wmnet [production]
14:22 <volans@cumin1001> START - Cookbook sre.hosts.decommission for hosts mc1027.eqiad.wmnet [production]
14:19 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:715492|Set permission of creating short url to everyone everywhere (T267921 T267925)]], Part II (duration: 00m 57s) [production]
14:17 <ladsgroup@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:715492|Set permission of creating short url to everyone everywhere (T267921 T267925)]], Part I (duration: 00m 59s) [production]
14:12 <moritzm> installing postgres 9.6 security updates [production]
14:05 <gehel> re-pooling wdqs1007, catched up on lag [production]
13:56 <jbond> update facter networking fact gerrit:715949 [production]