|
2019-04-11
§
|
| 12:15 |
<kartik@deploy1001> |
scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw] |
[production] |
| 12:01 |
<arturo> |
T151704 deploying oidentd |
[tools] |
| 11:54 |
<arturo> |
disable puppet in all hosts to deploy oidentd |
[tools] |
| 11:39 |
<zeljkof> |
EU SWAT finished |
[production] |
| 09:57 |
<elukey> |
roll restart druid-coordinator/overlord on druid100[4-6] to pick up new jvm settings |
[production] |
| 08:20 |
<moritzm> |
upgrading remaining job runners to HHVM 3.18.5+dfsg-1+wmf8+deb9u2 and wikidiff 1.8.1 (T203069) |
[production] |
| 06:33 |
<moritzm> |
uploaded jenkins 2.164.2 to apt.wikimedia.org (stretch-wikimedia / thirdparty/ci) |
[production] |
| 06:32 |
<moritzm> |
uploaded jenkins 2.164.2 to apt.wikimedia.org (jessie-wikimedia / thirdparty) |
[production] |
| 06:24 |
<moritzm> |
upgrading remaining API Servers to HHVM 3.18.5+dfsg-1+wmf8+deb9u2 and wikidiff 1.8.1 (T203069) |
[production] |
| 05:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove s3 ready only T219115 (duration: 00m 36s) |
[production] |
| 05:02 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Switchover s3 master eqiad from db1078 to db1075 T219115 (duration: 00m 36s) |
[production] |
| 05:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Set s3 on read-only T219115 (duration: 00m 37s) |
[production] |
| 05:00 |
<marostegui> |
Starting s3 failover from db1078 to db1075 - T219115 |
[production] |
| 04:32 |
<marostegui> |
Disable puppet on db1078 and db1075 T219115 |
[production] |
| 04:18 |
<marostegui> |
Start topology changes to move s3 slaves under db1075 T219115 |
[production] |
| 04:14 |
<marostegui> |
Disable GTID on s3 hosts - https://phabricator.wikimedia.org/T219115 |
[production] |
| 02:33 |
<andrewbogott> |
tools-paws-worker-1005, tools-paws-worker-1006 to eqiad1-r |
[tools] |
| 00:45 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.25/extensions/PageTriage/: UBN Fix for pageTriage and ORES T220649 (duration: 01m 04s) |
[production] |
| 00:12 |
<twentyafterfour> |
deploying phabricator upgrade |
[production] |
| 00:03 |
<andrewbogott> |
tools-paws-worker-1002, tools-paws-worker-1003 to eqiad1-r |
[tools] |
|
2019-04-10
§
|
| 23:58 |
<andrewbogott> |
moving tools-clushmaster-02, tools-elastic-03 and tools-paws-worker-1001 to eqiad1-r |
[tools] |
| 23:32 |
<James_F> |
Manually created REL1_33 branches for the core, vendor, and tarball extensions and skins. Eurgh. T220653 |
[releng] |
| 22:36 |
<James_F> |
Deleted faulty REL1_33 branches for the Timeless, Vector and Monobook skins; they were duplicates of the REL1_32 branches. |
[releng] |
| 21:35 |
<bstorm_> |
shut down webservice. It never runs at this time. It appears to need venv rebuilt. |
[tools.mediaviews-api] |
| 20:43 |
<urandom> |
decommissioning cassandra-c, restbase2007 -- T208087 |
[production] |
| 20:27 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Revert - Enabling api-request logging via eventgate-analytics for group1 wikis - T214080 (duration: 01m 00s) |
[production] |
| 20:27 |
<paladox> |
create operations/software/gerrit/plugins/MassBranchCreation repository |
[releng] |
| 19:48 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enabling api-request logging via eventgate-analytics for group1 wikis - T214080 (duration: 00m 59s) |
[production] |
| 19:42 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.33.0-wmf.25 refs T206679 (duration: 01m 48s) |
[production] |
| 19:40 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.25 refs T206679 |
[production] |
| 19:28 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.33.0-wmf.25 refs T206679 |
[production] |
| 19:26 |
<XioNoX> |
enable sampling on cr2-eqiad external links, outbound |
[production] |
| 19:23 |
<bstorm_> |
T220646 shut down webservice because it is just in a restart loop and cannot run |
[tools.dplbot] |
| 19:17 |
<twentyafterfour@deploy1001> |
Pruned MediaWiki: 1.33.0-wmf.20 [keeping static files] (duration: 02m 18s) |
[production] |
| 19:14 |
<ejegg> |
updated fundraising CiviCRM from d0e44a9e51 to 24b968b1f9 |
[production] |
| 19:08 |
<twentyafterfour@deploy1001> |
Pruned MediaWiki: 1.33.0-wmf.19 [keeping static files] (duration: 02m 22s) |
[production] |
| 18:52 |
<bstorm_> |
depooled and rebooted tools-sgeexec-0929 because systemd was in a weird state |
[tools] |
| 18:46 |
<bstorm_> |
depooled and rebooted tools-sgewebgrid-lighttpd-0913 because high load was caused by ancient lsof processes |
[tools] |
| 17:44 |
<twentyafterfour@deploy1001> |
Pruned MediaWiki: 1.33.0-wmf.18 [keeping static files] (duration: 02m 22s) |
[production] |
| 16:58 |
<chaomodus> |
restarted nagios-nrpe-server on proton1001 (it died due to OOM) |
[production] |
| 16:51 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=druid1004.eqiad.wmnet |
[production] |
| 16:01 |
<elukey> |
restart brokers on druid100[3-6] - locking after segments get deleted |
[production] |
| 15:46 |
<krinkle@deploy1001> |
Synchronized php-1.33.0-wmf.25/includes/parser/DateFormatter.php: Ib2b3fb315dc93b / T220563 (duration: 01m 00s) |
[production] |
| 15:28 |
<gilles@deploy1001> |
Synchronized php-1.33.0-wmf.25/includes/media/ThumbnailImage.php: T216499 Only apply high priority hint half the time (duration: 00m 59s) |
[production] |
| 15:26 |
<oblivian@deploy1001> |
Finished deploy [docker-pkg/deploy@605690c]: Upgrade to docker-pkg 2.0.0 everywhere (duration: 00m 21s) |
[production] |
| 15:26 |
<oblivian@deploy1001> |
Started deploy [docker-pkg/deploy@605690c]: Upgrade to docker-pkg 2.0.0 everywhere |
[production] |
| 15:24 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.25/extensions/Score/: UBN Revert Score changes that broke VE T220465 (duration: 01m 01s) |
[production] |
| 15:19 |
<oblivian@deploy1001> |
Finished deploy [docker-pkg/deploy@605690c]: Upgrade to docker-pkg 2.0.0 (duration: 00m 13s) |
[production] |
| 15:19 |
<oblivian@deploy1001> |
Started deploy [docker-pkg/deploy@605690c]: Upgrade to docker-pkg 2.0.0 |
[production] |
| 15:01 |
<fsero> |
pooled back mwdebug200[1,2] T219989 |
[production] |