6901-6950 of 10000 results (46ms)
2021-12-24 §
00:28 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1003.eqiad.wmnet with OS buster [production]
00:28 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1004.eqiad.wmnet with OS buster [production]
00:28 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster [production]
00:23 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster [production]
00:19 <legoktm> repooling mw1450 (forgot to after benchmarking finished) [production]
00:19 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
00:15 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
2021-12-23 §
21:41 <taavi@deploy1002> Finished deploy [horizon/deploy@ff82962] (dev): (no justification provided) (duration: 04m 06s) [production]
21:37 <taavi@deploy1002> Started deploy [horizon/deploy@ff82962] (dev): (no justification provided) [production]
18:17 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:14 <legoktm@deploy1002> Synchronized php-1.38.0-wmf.13/includes/changetags/ChangeTags.php: Disable querying the 'wikieditor' change tag temporarily (T298225) (duration: 00m 59s) [production]
18:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
18:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
17:50 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@363651a]: (no justification provided) (duration: 00m 06s) [production]
17:50 <mforns@deploy1002> Started deploy [airflow-dags/analytics@363651a]: (no justification provided) [production]
17:36 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED [production]
17:19 <volans@cumin2002> START - Cookbook sre.hosts.provision for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED [production]
17:14 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:10 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
17:04 <ejegg> updated standalone SmashPig deploy from 5a7d0c2c to 96c7b03e [production]
16:46 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:40 <volans@cumin1001> START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:26 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:13 <volans@cumin1001> START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:01 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:01 <volans@cumin1001> START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
11:34 <volans> upgraded spicerack to v1.1.1 on cumin1001,cumin2002 [production]
02:11 <ejegg> updated fundraising CiviCRM from aa90dd3a to 47dd67f2 [production]
00:04 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T297986 [production]
2021-12-22 §
23:01 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T297986 [production]
21:27 <volans> uploaded spicerack_1.1.1 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
18:42 <inflatador> T297735 removing/banning elastic1039 and elastic1043 from all EQIAD prod clusters [production]
18:16 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1088.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:11 <volans@cumin1001> START - Cookbook sre.hosts.provision for host elastic1088.mgmt.eqiad.wmnet with reboot policy FORCED [production]
17:51 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1085.mgmt.eqiad.wmnet with reboot policy FORCED [production]
17:45 <volans@cumin1001> START - Cookbook sre.hosts.provision for host elastic1085.mgmt.eqiad.wmnet with reboot policy FORCED [production]
17:42 <volans@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1085.mgmt.eqiad.wmnet with reboot policy FORCED [production]
17:42 <volans@cumin1001> START - Cookbook sre.hosts.provision for host elastic1085.mgmt.eqiad.wmnet with reboot policy FORCED [production]
17:35 <tzatziki> removing one file for legal compliance [production]
17:09 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
17:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
17:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
17:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
17:02 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
17:01 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
17:01 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
16:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
16:53 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
16:50 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]