|
2022-01-03
§
|
| 07:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2086.codfw.wmnet with reason: Maintenance |
[production] |
| 07:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2086.codfw.wmnet with reason: Maintenance |
[production] |
| 07:17 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 07:13 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 07:13 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 07:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 07:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 07:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 07:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 07:02 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:750831|Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708)]], Part I (duration: 01m 20s) |
[production] |
| 07:01 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 07:00 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:750831|Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708)]], Part I (duration: 00m 58s) |
[production] |
| 06:53 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance |
[production] |
| 06:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance |
[production] |
| 04:21 |
<Amir1> |
start of running populating actor in revision table on rest of sections. It will take two months to finish (T275246) |
[production] |
|
2021-12-28
§
|
| 11:27 |
<godog> |
powercycle ms-be1059 -- was powered down |
[production] |
| 02:39 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 02:35 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 02:35 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 02:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 02:21 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 02:17 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 02:17 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 02:14 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
|
2021-12-24
§
|
| 20:08 |
<mforns@deploy1002> |
Finished deploy [airflow-dags/analytics@e282d2d]: (no justification provided) (duration: 00m 06s) |
[production] |
| 20:08 |
<mforns@deploy1002> |
Started deploy [airflow-dags/analytics@e282d2d]: (no justification provided) |
[production] |
| 04:20 |
<legoktm> |
depooled cp2029 now that it's up |
[production] |
| 04:16 |
<legoktm> |
powercycling cp2029 via mgmt |
[production] |
| 00:57 |
<ejegg> |
updated fundraising CiviCRM from 47dd67f2 to aaceb4ab |
[production] |
| 00:28 |
<cmjohnson@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1003.eqiad.wmnet with OS buster |
[production] |
| 00:28 |
<cmjohnson@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1004.eqiad.wmnet with OS buster |
[production] |
| 00:28 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster |
[production] |
| 00:23 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster |
[production] |
| 00:19 |
<legoktm> |
repooling mw1450 (forgot to after benchmarking finished) |
[production] |
| 00:19 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 00:15 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
|
2021-12-23
§
|
| 21:41 |
<taavi@deploy1002> |
Finished deploy [horizon/deploy@ff82962] (dev): (no justification provided) (duration: 04m 06s) |
[production] |
| 21:37 |
<taavi@deploy1002> |
Started deploy [horizon/deploy@ff82962] (dev): (no justification provided) |
[production] |
| 18:17 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 18:14 |
<legoktm@deploy1002> |
Synchronized php-1.38.0-wmf.13/includes/changetags/ChangeTags.php: Disable querying the 'wikieditor' change tag temporarily (T298225) (duration: 00m 59s) |
[production] |
| 18:14 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 18:14 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 18:13 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 17:50 |
<mforns@deploy1002> |
Finished deploy [airflow-dags/analytics@363651a]: (no justification provided) (duration: 00m 06s) |
[production] |
| 17:50 |
<mforns@deploy1002> |
Started deploy [airflow-dags/analytics@363651a]: (no justification provided) |
[production] |
| 17:36 |
<volans@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 17:19 |
<volans@cumin2002> |
START - Cookbook sre.hosts.provision for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 17:14 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:10 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |