|
2019-06-06
§
|
| 08:13 |
<hashar> |
Reloading Zuul for I764972711843645afd00e196a3bedd17730b4cbe which drops mwselenium-quibble-docker from Wikibase |
[releng] |
| 07:35 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1121 after upgrade T224852 (duration: 00m 53s) |
[production] |
| 07:20 |
<marostegui> |
Stop MySQL on db1121 for upgrade, this will generate lag on labs hosts for s6 - T224852 |
[production] |
| 07:16 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Promote db2046 to s6 master as db2039 will be decommissioned T221533 (duration: 00m 55s) |
[production] |
| 06:38 |
<elukey> |
re-run refine_sanitize_eventlogging_analytics_immediate with since = 48 in the .properties file (manually added) |
[analytics] |
| 06:31 |
<marostegui> |
Start topology changes on s6 codfw to promote db2046 as master - T221533 |
[production] |
| 06:23 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1121 for upgrade T224852 (duration: 00m 55s) |
[production] |
| 06:15 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1091 after getting its BBU replaced (duration: 00m 54s) |
[production] |
| 06:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 01m 01s) |
[production] |
| 05:47 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 00m 55s) |
[production] |
| 05:41 |
<marostegui> |
Upgrade MySQL on s6 codfw hosts in preparation for s6 codfw master failover - T221533 |
[production] |
| 05:36 |
<elukey> |
chown analytics:analytics /wmf/data/event_sanitized/{CentralNoticeTiming,LayoutJank,EventTiming,ElementTiming} (new directories created with yarn:analytics) |
[analytics] |
| 05:32 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after getting its BBU replaced (duration: 00m 55s) |
[production] |
| 05:18 |
<marostegui> |
Remove db2042 from tendril and zarcillo T225090 |
[production] |
| 05:18 |
<marostegui> |
Remove db2042 from tendril and zarcillo |
[production] |
| 05:14 |
<marostegui> |
Stop MySQL on db2042 to copy its content to dbprov2001 as a temporary backup - T225090 |
[production] |
| 05:11 |
<marostegui> |
Disable notifications db2042 - T225090 |
[production] |
| 05:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1091 after getting its BBU replaced T225060 (duration: 00m 56s) |
[production] |
|
2019-06-05
§
|
| 22:50 |
<Krenair> |
Added cloudcontrol1004 IP to match cloudcontrol1003 rule in 'proxy' security group rules for port 5668 T225168 |
[project-proxy] |
| 22:44 |
<Krenair> |
Updating 'proxy' security group rules for port 5668 to remove decommissioned IP - 208.80.154.147 californium T189921 |
[project-proxy] |
| 22:36 |
<Krenair> |
Updating 'proxy' security group rules for port 5668 to remove decommissioned IPs - 208.80.154.136 silver, 208.80.155.117 labs-ns0, 208.80.152.32 virt0 (!), 208.80.153.48 labtestservices2001, 208.80.154.92 labcontrol1001 |
[project-proxy] |
| 22:15 |
<chaomodus> |
restarting gerrit on cobalt due to it being down (seems like Java out of heap space) |
[production] |
| 22:14 |
<Krenair> |
Per jeh's investigation, added cloudservices1004 IP to match cloudservices1003 rule in 'proxy' security group rules for port 5668 |
[project-proxy] |
| 20:59 |
<mforns> |
finished deployment of analytics/refinery up to 0660e70153dec892ae20bee7119a72cc17e8ec87 |
[analytics] |
| 20:43 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@0660e70]: deploying analytics/refinery up to 0660e70153dec892ae20bee7119a72cc17e8ec87 (duration: 19m 30s) |
[production] |
| 20:39 |
<reedy@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: Turn off some FR config T225138 (duration: 00m 54s) |
[production] |
| 20:25 |
<akosiaris@deploy1001> |
scap-helm blubberoid finished |
[production] |
| 20:25 |
<akosiaris@deploy1001> |
scap-helm blubberoid cluster codfw completed |
[production] |
| 20:25 |
<akosiaris@deploy1001> |
scap-helm blubberoid cluster eqiad completed |
[production] |
| 20:25 |
<akosiaris@deploy1001> |
scap-helm blubberoid upgrade -f blubberoid-values.yaml production stable/blubberoid [namespace: blubberoid, clusters: eqiad,codfw] |
[production] |
| 20:23 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@0660e70]: deploying analytics/refinery up to 0660e70153dec892ae20bee7119a72cc17e8ec87 |
[production] |
| 20:20 |
<mforns> |
starting deployment of analytics/refinery up to 0660e70153dec892ae20bee7119a72cc17e8ec87 |
[analytics] |
| 20:02 |
<bd808> |
Restarted bot to pick up config change adding #wikimedia-codehealth |
[tools.stashbot] |
| 19:57 |
<hashar> |
contint1001: docker container prune -f && docker image prune -f # reclaimed 166 MB and 3.4 GB |
[production] |
| 19:50 |
<wikibugs> |
Updated channels.yaml to: 445187388f45f0a9dd7d038cdfe41a5990e373a8 Add #wikimedia-codehealth channel |
[tools.wikibugs] |
| 19:48 |
<marostegui> |
Check data consistency on db1091 against db1135 - T225060 |
[production] |
| 19:45 |
<reedy@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: T225115 (duration: 00m 54s) |
[production] |
| 19:34 |
<andrewbogott> |
moving deployment-imagescaler03 to cloudvirt1029 |
[releng] |
| 19:34 |
<andrewbogott> |
moving deployment-imagescaler03 to cloudvirt1029 |
[deployment-prep] |
| 18:33 |
<andrewbogott> |
repooled tools-sgeexec-0921 and tools-sgeexec-0929 |
[tools] |
| 18:24 |
<andrewbogott> |
moving maps-puppetmaster to cloudvirt1029 |
[maps] |
| 18:20 |
<mforns> |
finished deployment of analytics/refinery/source v0.0.91 |
[analytics] |
| 18:16 |
<andrewbogott> |
depooling and moving tools-sgeexec-0921 and tools-sgeexec-0929 |
[tools] |
| 18:00 |
<mforns> |
starting deployment of analytics/refinery/source v0.0.91 |
[analytics] |
| 17:36 |
<marostegui> |
Start replication db1091 - T225060 |
[production] |
| 17:35 |
<James_F> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/514547 |
[releng] |
| 17:32 |
<marostegui> |
Start MySQL with replication stopped on db1091 - T225060 |
[production] |
| 16:29 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Revert user-blocks-change to use eventbus and old schema - T211248 (duration: 00m 54s) |
[production] |
| 16:22 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: use eventgate-main for 2 events on all wikis - T211248 (duration: 00m 55s) |
[production] |
| 16:11 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add wgEventServiceStreamConfig and switch 2 topics in group0 T222822 (duration: 00m 56s) |
[production] |