|
2019-11-20
ยง
|
| 17:24 |
<mobrovac@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'citoid' for release 'production' . |
[production] |
| 17:21 |
<mobrovac@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'citoid' for release 'production' . |
[production] |
| 17:19 |
<mobrovac@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'citoid' for release 'staging' . |
[production] |
| 17:18 |
<andrewbogott> |
upgrading pdns to version 4 on cloudservices1003 |
[production] |
| 17:06 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:04 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:03 |
<andrewbogott> |
upgrading pdns to version 4 on cloudvirt1004 T210715 |
[production] |
| 16:58 |
<andrewbogott> |
disabling puppet on cloudvirt1003 and 1004 for T210715 |
[production] |
| 16:55 |
<moritzm> |
installing rpcbind bugfix updates from buster 10.2 point release |
[production] |
| 16:43 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:40 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:23 |
<ema> |
depool cp2023 and reimage as text_ats T227432 |
[production] |
| 16:14 |
<ema> |
pool cp2019 with ATS backend T227432 |
[production] |
| 16:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3314 after compression', diff saved to https://phabricator.wikimedia.org/P9695 and previous config saved to /var/cache/conftool/dbconfig/20191120-160813-marostegui.json |
[production] |
| 16:03 |
<gehel> |
depool wdqs1004 to allow catching up on lag - T238229 |
[production] |
| 15:42 |
<mobrovac@deploy1001> |
Synchronized wmf-config/LabsServices.php: [BETA-ONLY] Switch Flow to use Parsoid/PHP - T229078 (duration: 00m 52s) |
[production] |
| 15:40 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:38 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:36 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: RESYNC T221774 - wgWikidataOrgQueryServiceMaxLagFactor 180 [[gerrit:552069]] (duration: 00m 52s) |
[production] |
| 15:19 |
<ema> |
depool cp2019 and reimage as text_ats T227432 |
[production] |
| 15:08 |
<gehel> |
reset LVS weight for wdqs public eqiad to 10 |
[production] |
| 15:05 |
<effie> |
Enable puppet on mw* |
[production] |
| 14:52 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T221774 - wgWikidataOrgQueryServiceMaxLagFactor 180 [[gerrit:552069]] (duration: 00m 52s) |
[production] |
| 14:50 |
<addshore@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Wikidata.org: T221774 - Wikidata.org extension (use altered lag, not raw lag) [[gerrit:552072]] (duration: 00m 53s) |
[production] |
| 14:49 |
<ema> |
pool cp2016 with ATS backend T227432 |
[production] |
| 14:47 |
<effie> |
disable puppet on all mw* servers |
[production] |
| 14:27 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:24 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:06 |
<ema> |
depool cp2016 and reimage as text_ats T227432 |
[production] |
| 13:32 |
<godog> |
updated puppet compiler facts on compiler100* hosts |
[production] |
| 12:43 |
<ema> |
pool cp2013 with ATS backend T227432 |
[production] |
| 12:27 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:25 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:08 |
<ema> |
depool cp2013 and reimage as text_ats T227432 |
[production] |
| 11:58 |
<ema> |
pool cp2012 with ATS backend T227432 |
[production] |
| 11:55 |
<Urbanecm> |
EU SWAT done |
[production] |
| 11:55 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 2b13fbe: [rowiki] Enable deleterevision for patrollers (T234051) (duration: 00m 52s) |
[production] |
| 11:46 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 51ecd71: Partial cleanup of InitializeSettings (T231178) (duration: 00m 52s) |
[production] |
| 11:42 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:40 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:38 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: f847380: Set namespace alias for Index: (NS 102/103) for elwikisource (T237253) (duration: 00m 54s) |
[production] |
| 11:36 |
<urbanecm@deploy1001> |
Finished scap: SWAT: 44ec4e4: e1baf0e: 3c02aa7: Namespace changes (duration: 06m 15s) |
[production] |
| 11:30 |
<urbanecm@deploy1001> |
Started scap: SWAT: 44ec4e4: e1baf0e: 3c02aa7: Namespace changes |
[production] |
| 11:27 |
<ema> |
cp2010: ats-backend-restart to clear backend restart alert |
[production] |
| 11:21 |
<ema> |
depool cp2012 and reimage as text_ats T227432 |
[production] |
| 11:15 |
<ema> |
pool cp2010 with ATS backend T227432 |
[production] |
| 10:54 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:52 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:36 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@daa7808]: Revert switching test2.wp to Parsoid/JS - T238716 (duration: 13m 56s) |
[production] |
| 10:34 |
<ema> |
depool cp2010 and reimage as text_ats T227432 |
[production] |