|
2020-01-23
ยง
|
| 16:56 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:51 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 16:27 |
<vgutierrez> |
depool cp4032 and reimage as buster - T242093 |
[production] |
| 16:26 |
<vgutierrez> |
pooling cp4026 running buster - T242093 |
[production] |
| 16:02 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/extensions/Wikibase/data-access/src/EntitySourceDefinitions.php: [[gerrit:566721|EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities (T242415 T214557)]] (duration: 01m 08s) |
[production] |
| 16:00 |
<rlazarus> |
Starting etcd main cluster switchover from codfw to eqiad |
[production] |
| 15:45 |
<vgutierrez> |
restarting high-traffic1 && high-traffic2 primary LVSs - T236120 T238625 |
[production] |
| 15:32 |
<vgutierrez> |
restarting secondary LVSs - T236120 T238625 |
[production] |
| 15:31 |
<halfak> |
deploying ores 283f627a |
[releng] |
| 15:31 |
<halfak> |
deploying ores 283f627a |
[deployment-prep] |
| 15:22 |
<moritzm> |
mask uwsgi.service on debmonitor2001 T222874 |
[production] |
| 15:06 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: name=cp4026.ulsfo.wmnet,service=nginx |
[production] |
| 14:39 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=ats-tls,name=cp4026.ulsfo.wmnet |
[production] |
| 14:17 |
<marostegui> |
Remove wikiadmin2 user from codfw x1 hosts - T243512 |
[production] |
| 13:34 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:32 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:19 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:19 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:50 |
<Amir1> |
EU SWAT is done |
[production] |
| 12:48 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:566716|Set EntitySourceBasedFederation true for testwiki (T243395)]] (duration: 01m 06s) |
[production] |
| 12:47 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:566716|Set EntitySourceBasedFederation true for testwiki (T243395)]] (duration: 01m 05s) |
[production] |
| 12:46 |
<Urbanecm> |
Run renameRestrictions.php 'autopatrol' 'editautopatrolprotected' for all Serbian wikis (T230103) |
[production] |
| 12:44 |
<Urbanecm> |
mwscript renameRestrictions.php --wiki=hewiki 'autopatrol' 'editautopatrolprotected' (T230103) |
[production] |
| 12:44 |
<Urbanecm> |
mwscript renameRestrictions.php --wiki=etwiki 'autopatrol' 'editautopatrolprotected' (T230103) |
[production] |
| 12:40 |
<urbanecm@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: SWAT: 0c2fb70: Use editautopatrolprotected right for pages protected for autopatrollers (3/3; T230103) (duration: 01m 05s) |
[production] |
| 12:39 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: 0c2fb70: Use editautopatrolprotected right for pages protected for autopatrollers (2/3; T230103) (duration: 01m 08s) |
[production] |
| 12:35 |
<Urbanecm> |
mwscript renameRestrictions.php --wiki=ckbwiki 'autopatrol' 'editautopatrolprotected' (T230103) |
[production] |
| 12:33 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 0c2fb70: Use editautopatrolprotected right for pages protected for autopatrollers; fixing broken cache (T230103) (duration: 01m 04s) |
[production] |
| 12:31 |
<twentyafterfour> |
Deploying hotfix for T243479, restarting php7.3-fpm on phab1003 |
[production] |
| 12:31 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 0c2fb70: Use editautopatrolprotected right for pages protected for autopatrollers (T230103) (duration: 01m 06s) |
[production] |
| 12:15 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:566701|Set useEntitySourceBasedFederation to true for Wikidata (T241972)]] (duration: 01m 04s) |
[production] |
| 12:14 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:566701|Set useEntitySourceBasedFederation to true for Wikidata (T241972)]] (duration: 01m 06s) |
[production] |
| 12:10 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:566123|Move CX out of beta for af, is, lv and ne WPs (T242011 T242012 T242014 T242016)]] (duration: 01m 05s) |
[production] |
| 12:08 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:566123|Move CX out of beta for af, is, lv and ne WPs (T242011 T242012 T242014 T242016)]] (duration: 01m 08s) |
[production] |
| 11:37 |
<jbond42> |
updating order in resolve search list https://gerrit.wikimedia.org/r/c/operations/puppet/+/566567 |
[production] |
| 10:25 |
<vgutierrez> |
depooling and reimaging cp4026 as buster - T242093 |
[production] |
| 09:13 |
<moritzm> |
installing xen updates (only pulled in via deps, otherwise unused) |
[production] |
| 08:45 |
<marostegui> |
Stop mysql on es2024 to "clone" es2025 - T243052 |
[production] |
| 06:05 |
<marostegui> |
Remove partitions from db1097:3314 - T239453 |
[production] |
| 06:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1097:3314 - T239453', diff saved to https://phabricator.wikimedia.org/P10248 and previous config saved to /var/cache/conftool/dbconfig/20200123-060308-marostegui.json |
[production] |
| 05:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3314 - T239453', diff saved to https://phabricator.wikimedia.org/P10247 and previous config saved to /var/cache/conftool/dbconfig/20200123-055919-marostegui.json |
[production] |
| 05:55 |
<marostegui> |
Compress some tables on db1124:3318, this might generate lag on s8 labs - T232446 |
[production] |
| 05:15 |
<bd808> |
Building tools-elastic-04 |
[tools] |
| 04:39 |
<bd808> |
wmcs-openstack quota set --instances 192 |
[tools] |
| 04:36 |
<bd808> |
wmcs-openstack quota set --cores 768 --ram 1536000 |
[tools] |
| 03:14 |
<bd808> |
Demoted projectadmins not listed in the "roots" sudoer policy to project members just to avoid random confusion |
[toolsbeta] |
| 03:06 |
<bd808> |
Added legoktm to "roots" sudoer policy |
[toolsbeta] |
| 02:53 |
<bd808> |
Added legoktm as project admin |
[toolsbeta] |
| 02:00 |
<legoktm> |
shutting down and deleting temporary codesearch5 instance |
[codesearch] |
| 01:59 |
<legoktm> |
shutting down old codesearch4 instance (but not deleting yet) |
[codesearch] |