9601-9650 of 10000 results (58ms)
2022-01-03 §
07:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2087.codfw.wmnet with reason: Maintenance [production]
07:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2087.codfw.wmnet with reason: Maintenance [production]
07:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2086.codfw.wmnet with reason: Maintenance [production]
07:23 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2086.codfw.wmnet with reason: Maintenance [production]
07:17 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
07:13 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
07:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
07:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
07:04 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
07:03 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
07:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
07:02 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:750831|Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708)]], Part I (duration: 01m 20s) [production]
07:01 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
07:00 <ladsgroup@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:750831|Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708)]], Part I (duration: 00m 58s) [production]
06:53 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance [production]
06:53 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance [production]
04:21 <Amir1> start of running populating actor in revision table on rest of sections. It will take two months to finish (T275246) [production]
2021-12-29 §
10:30 <elukey> kill tcpdump process on kubestagemaster1001 (kept a big pcap file opened that kept growing) [production]
2021-12-28 §
11:27 <godog> powercycle ms-be1059 -- was powered down [production]
02:39 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:35 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
02:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
02:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:17 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
02:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
2021-12-24 §
20:08 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@e282d2d]: (no justification provided) (duration: 00m 06s) [production]
20:08 <mforns@deploy1002> Started deploy [airflow-dags/analytics@e282d2d]: (no justification provided) [production]
04:20 <legoktm> depooled cp2029 now that it's up [production]
04:16 <legoktm> powercycling cp2029 via mgmt [production]
00:57 <ejegg> updated fundraising CiviCRM from 47dd67f2 to aaceb4ab [production]
00:28 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1003.eqiad.wmnet with OS buster [production]
00:28 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1004.eqiad.wmnet with OS buster [production]
00:28 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster [production]
00:23 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster [production]
00:19 <legoktm> repooling mw1450 (forgot to after benchmarking finished) [production]
00:19 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
00:15 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
2021-12-23 §
21:41 <taavi@deploy1002> Finished deploy [horizon/deploy@ff82962] (dev): (no justification provided) (duration: 04m 06s) [production]
21:37 <taavi@deploy1002> Started deploy [horizon/deploy@ff82962] (dev): (no justification provided) [production]
18:17 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:14 <legoktm@deploy1002> Synchronized php-1.38.0-wmf.13/includes/changetags/ChangeTags.php: Disable querying the 'wikieditor' change tag temporarily (T298225) (duration: 00m 59s) [production]
18:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
18:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
17:50 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@363651a]: (no justification provided) (duration: 00m 06s) [production]
17:50 <mforns@deploy1002> Started deploy [airflow-dags/analytics@363651a]: (no justification provided) [production]
17:36 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED [production]
17:19 <volans@cumin2002> START - Cookbook sre.hosts.provision for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED [production]