1951-2000 of 10000 results (68ms)
2022-09-09 §
10:31 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 09s) [production]
10:31 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b9be20d]: (no justification provided) [production]
10:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34328 and previous config saved to /var/cache/conftool/dbconfig/20220909-101830-root.json [production]
10:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34327 and previous config saved to /var/cache/conftool/dbconfig/20220909-100324-root.json [production]
09:53 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 09s) [production]
09:53 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b9be20d]: (no justification provided) [production]
09:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34326 and previous config saved to /var/cache/conftool/dbconfig/20220909-094819-root.json [production]
09:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34325 and previous config saved to /var/cache/conftool/dbconfig/20220909-093314-root.json [production]
09:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34323 and previous config saved to /var/cache/conftool/dbconfig/20220909-091809-root.json [production]
08:59 <cgoubert@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wtp[1029-1033].eqiad.wmnet [production]
08:59 <cgoubert@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:56 <cgoubert@cumin1001> START - Cookbook sre.dns.netbox [production]
08:37 <cgoubert@cumin1001> START - Cookbook sre.hosts.decommission for hosts wtp[1029-1033].eqiad.wmnet [production]
08:32 <dcausse> rebuilding all completion indices in elastic@codfw [production]
08:16 <dcausse> restarting on blazegraph on wdqs2002 (BlazegraphFreeAllocatorsDecreasingRapidly) [production]
08:12 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
08:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2106 (T314041)', diff saved to https://phabricator.wikimedia.org/P34322 and previous config saved to /var/cache/conftool/dbconfig/20220909-081103-ladsgroup.json [production]
08:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance [production]
08:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance [production]
08:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1121 (T314041)', diff saved to https://phabricator.wikimedia.org/P34321 and previous config saved to /var/cache/conftool/dbconfig/20220909-081042-ladsgroup.json [production]
08:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
08:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
08:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
08:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
08:06 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
08:06 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
08:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1106 (T312863)', diff saved to https://phabricator.wikimedia.org/P34320 and previous config saved to /var/cache/conftool/dbconfig/20220909-080609-ladsgroup.json [production]
08:06 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
08:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
08:05 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance [production]
08:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance [production]
07:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1157 (T312863)', diff saved to https://phabricator.wikimedia.org/P34319 and previous config saved to /var/cache/conftool/dbconfig/20220909-074710-ladsgroup.json [production]
07:47 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
07:46 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
07:46 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
07:14 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1007.eqiad.wmnet [production]
07:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1137 for upgrade', diff saved to https://phabricator.wikimedia.org/P34318 and previous config saved to /var/cache/conftool/dbconfig/20220909-071255-root.json [production]
07:10 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host an-tool1007.eqiad.wmnet [production]
05:44 <marostegui> dbmaint s8 wikidatawiki eqiad T317349 [production]
05:43 <marostegui> dbmaint s3 testwikidatawiki eqiad T317349 [production]
05:42 <marostegui> dbmaint s4 commonswiki eqiad T317349 [production]
05:41 <marostegui> dbmaint s4 testcommonswiki eqiad T317349 [production]
05:19 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
05:18 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
05:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
05:11 <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: cirrus: Switch all wikis from completion suggester to prefix search, yesterdays completion index builds in codfw weren't all succesfull and users are getting incomplete results (duration: 04m 01s) [production]
05:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
00:09 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 22s) [production]
00:08 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b9be20d]: (no justification provided) [production]
2022-09-08 §
23:56 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 27s) [production]