1801-1850 of 10000 results (41ms)
2022-01-03 §
07:00 <ladsgroup@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:750831|Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708)]], Part I (duration: 00m 58s) [production]
06:53 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance [production]
06:53 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance [production]
04:21 <Amir1> start of running populating actor in revision table on rest of sections. It will take two months to finish (T275246) [production]
2021-12-29 §
10:30 <elukey> kill tcpdump process on kubestagemaster1001 (kept a big pcap file opened that kept growing) [production]
2021-12-28 §
11:27 <godog> powercycle ms-be1059 -- was powered down [production]
02:39 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:35 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
02:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
02:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:17 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
02:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
2021-12-24 §
20:08 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@e282d2d]: (no justification provided) (duration: 00m 06s) [production]
20:08 <mforns@deploy1002> Started deploy [airflow-dags/analytics@e282d2d]: (no justification provided) [production]
04:20 <legoktm> depooled cp2029 now that it's up [production]
04:16 <legoktm> powercycling cp2029 via mgmt [production]
00:57 <ejegg> updated fundraising CiviCRM from 47dd67f2 to aaceb4ab [production]
00:28 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1003.eqiad.wmnet with OS buster [production]
00:28 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1004.eqiad.wmnet with OS buster [production]
00:28 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster [production]
00:23 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster [production]
00:19 <legoktm> repooling mw1450 (forgot to after benchmarking finished) [production]
00:19 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
00:15 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
2021-12-23 §
21:41 <taavi@deploy1002> Finished deploy [horizon/deploy@ff82962] (dev): (no justification provided) (duration: 04m 06s) [production]
21:37 <taavi@deploy1002> Started deploy [horizon/deploy@ff82962] (dev): (no justification provided) [production]
18:17 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:14 <legoktm@deploy1002> Synchronized php-1.38.0-wmf.13/includes/changetags/ChangeTags.php: Disable querying the 'wikieditor' change tag temporarily (T298225) (duration: 00m 59s) [production]
18:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
18:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
17:50 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@363651a]: (no justification provided) (duration: 00m 06s) [production]
17:50 <mforns@deploy1002> Started deploy [airflow-dags/analytics@363651a]: (no justification provided) [production]
17:36 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED [production]
17:19 <volans@cumin2002> START - Cookbook sre.hosts.provision for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED [production]
17:14 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:10 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
17:04 <ejegg> updated standalone SmashPig deploy from 5a7d0c2c to 96c7b03e [production]
16:46 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:40 <volans@cumin1001> START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:26 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:13 <volans@cumin1001> START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:01 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:01 <volans@cumin1001> START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
11:34 <volans> upgraded spicerack to v1.1.1 on cumin1001,cumin2002 [production]
02:11 <ejegg> updated fundraising CiviCRM from aa90dd3a to 47dd67f2 [production]
00:04 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T297986 [production]
2021-12-22 §
23:01 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T297986 [production]