|
2020-01-23
ยง
|
| 18:59 |
<mutante> |
ganeti1003 - creating new VM etherpad1002.eqiad.wmnet with 1GB RAM and 10GB disk, row C, private link (T243475) |
[production] |
| 18:58 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 18:54 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 18:47 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 18:40 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Stop setting wgWikimediaMessagesPartialBlockBanner, never read T240300 (duration: 01m 06s) |
[production] |
| 18:35 |
<rlazarus> |
etcd main cluster switchover complete, eqiad is now read-write |
[production] |
| 18:28 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 18:27 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 18:22 |
<vgutierrez> |
pooling cp4032 running buster - T242093 |
[production] |
| 18:15 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 18:05 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 18:05 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 18:03 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 18:03 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 18:01 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 18:01 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 17:59 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 17:59 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 17:53 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 17:53 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 17:52 |
<_joe_> |
running systemctl reset-failed on conf1005 to clear useless alerts |
[production] |
| 17:33 |
<marostegui> |
Poweroff db2085:3311 and db2085:3318 for maintenance - T243148 |
[production] |
| 17:33 |
<jforrester@deploy1001> |
Synchronized static/images/project-logos: [trwiki] Tweak logo versions T242977 (duration: 01m 07s) |
[production] |
| 17:00 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 16:59 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 16:58 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:56 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:51 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 16:27 |
<vgutierrez> |
depool cp4032 and reimage as buster - T242093 |
[production] |
| 16:26 |
<vgutierrez> |
pooling cp4026 running buster - T242093 |
[production] |
| 16:02 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/extensions/Wikibase/data-access/src/EntitySourceDefinitions.php: [[gerrit:566721|EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities (T242415 T214557)]] (duration: 01m 08s) |
[production] |
| 16:00 |
<rlazarus> |
Starting etcd main cluster switchover from codfw to eqiad |
[production] |
| 15:45 |
<vgutierrez> |
restarting high-traffic1 && high-traffic2 primary LVSs - T236120 T238625 |
[production] |
| 15:32 |
<vgutierrez> |
restarting secondary LVSs - T236120 T238625 |
[production] |
| 15:22 |
<moritzm> |
mask uwsgi.service on debmonitor2001 T222874 |
[production] |
| 15:06 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: name=cp4026.ulsfo.wmnet,service=nginx |
[production] |
| 14:39 |
<vgutierrez@puppetmaster1001> |
conftool action : set/weight=1; selector: service=ats-tls,name=cp4026.ulsfo.wmnet |
[production] |
| 14:17 |
<marostegui> |
Remove wikiadmin2 user from codfw x1 hosts - T243512 |
[production] |
| 13:34 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:32 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:19 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:19 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:50 |
<Amir1> |
EU SWAT is done |
[production] |
| 12:48 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:566716|Set EntitySourceBasedFederation true for testwiki (T243395)]] (duration: 01m 06s) |
[production] |
| 12:47 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:566716|Set EntitySourceBasedFederation true for testwiki (T243395)]] (duration: 01m 05s) |
[production] |
| 12:46 |
<Urbanecm> |
Run renameRestrictions.php 'autopatrol' 'editautopatrolprotected' for all Serbian wikis (T230103) |
[production] |
| 12:44 |
<Urbanecm> |
mwscript renameRestrictions.php --wiki=hewiki 'autopatrol' 'editautopatrolprotected' (T230103) |
[production] |
| 12:44 |
<Urbanecm> |
mwscript renameRestrictions.php --wiki=etwiki 'autopatrol' 'editautopatrolprotected' (T230103) |
[production] |
| 12:40 |
<urbanecm@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: SWAT: 0c2fb70: Use editautopatrolprotected right for pages protected for autopatrollers (3/3; T230103) (duration: 01m 05s) |
[production] |
| 12:39 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: 0c2fb70: Use editautopatrolprotected right for pages protected for autopatrollers (2/3; T230103) (duration: 01m 08s) |
[production] |