|
2021-04-02
§
|
| 15:48 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 15:45 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 15:45 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 15:41 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 14:35 |
<jiji@cumin1001> |
conftool action : set/weight=20; selector: cluster=jobrunner,name=mw133[7-8].eqiad.wmnet |
[production] |
| 14:34 |
<jiji@cumin1001> |
conftool action : set/weight=20; selector: cluster=videoscaler,name=mw133[5-6].eqiad.wmnet |
[production] |
| 14:32 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw133[5-6].eqiad.wmnet |
[production] |
| 14:31 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw133[7-8].eqiad.wmnet |
[production] |
| 14:30 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-coord1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:29 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw1111.eqiad.wmnet |
[production] |
| 14:28 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-coord1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:20 |
<Urbanecm> |
Start server-side upload for 3 video files (T279060, T279061, T279062) |
[production] |
| 14:09 |
<Urbanecm> |
Start server-side upload for 3 video files (T279138, T279137, T279136) |
[production] |
| 13:42 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.37 |
[production] |
| 13:14 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 13:12 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 13:11 |
<reedy@deploy1002> |
Synchronized php-1.36.0-wmf.37/load.php: T278579 (duration: 00m 58s) |
[production] |
| 13:10 |
<reedy@deploy1002> |
Synchronized php-1.36.0-wmf.37/includes/OutputHandler.php: T278579 (duration: 00m 57s) |
[production] |
| 13:08 |
<reedy@deploy1002> |
Synchronized php-1.36.0-wmf.37/includes/MediaWiki.php: T278579 (duration: 00m 58s) |
[production] |
| 11:46 |
<Urbanecm> |
correction: Start server-side upload for 3 video files (T279079, T279080, T279104) |
[production] |
| 11:45 |
<Urbanecm> |
Start server-side upload for 3 images (T279079, T279080, T279104) |
[production] |
| 10:54 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 10:52 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 10:14 |
<akosiaris@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 10:14 |
<akosiaris@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 10:12 |
<akosiaris@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 10:12 |
<akosiaris@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 10:11 |
<akosiaris@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 10:11 |
<akosiaris@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 10:07 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: Rollback group0 wikis to 1.36.0-wmf.36 - T278343 |
[production] |
| 09:45 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: Revert group1 and group2 wikis to 1.36.0-wmf.36 - T278343 |
[production] |
| 09:44 |
<hashar@deploy1002> |
sync-wikiversions aborted: Revert group1 and group2 wikis to 1.36.0-wmf.36 (duration: 00m 01s) |
[production] |
| 09:06 |
<dcausse> |
remove dumps from wdqs1009 to free disk space |
[production] |
| 07:33 |
<effie> |
powercycle an-worker1080 |
[production] |
| 07:28 |
<elukey> |
manual fix for an-worker1080's interface in netbox (xe-4/0/11), moved by mistake to public-1b |
[production] |
| 03:54 |
<dwisehaupt> |
replication user on fundraising db set to require ssl for connections at the mysql user level. db updated on frdb1004 and verified on a set of hosts |
[production] |
| 03:16 |
<dwisehaupt> |
replication user on payments db set to require ssl for connections at the mysql user level. db updated on payments1001 and verified on a set of hosts |
[production] |
|
2021-04-01
§
|
| 23:32 |
<thcipriani@deploy1002> |
Synchronized php-1.36.0-wmf.37/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: Backport: [[gerrit:676350|Revert "Turn on glent m1 AB test"]] T262612 (duration: 00m 58s) |
[production] |
| 23:28 |
<thcipriani> |
reset /srv/mediawiki-staging/php-1.36.0-wmf.37/extensions/TimedMediaHandler to 1be781d (HEAD of wmf/1.36.0-wmf.37 -- from HEAD of master 49f417) |
[production] |
| 23:12 |
<thcipriani@deploy1002> |
Synchronized wmf-config/logos.php: Backport: Part III [[gerrit:676451|Add hi-res version of mediawiki.org logos]] T268230 (duration: 00m 57s) |
[production] |
| 23:10 |
<thcipriani@deploy1002> |
Synchronized logos: Backport: Part II [[gerrit:676451|Add hi-res version of mediawiki.org logos]] T268230 (duration: 00m 57s) |
[production] |
| 23:08 |
<thcipriani@deploy1002> |
Synchronized static: Backport: Part I [[gerrit:676451|Add hi-res version of mediawiki.org logos]] T268230 (duration: 00m 59s) |
[production] |
| 22:53 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2248.codfw.wmnet |
[production] |
| 22:50 |
<twentyafterfour@deploy1002> |
Finished deploy [releng/phatality@27ddd0b]: deploy phatality (duration: 00m 13s) |
[production] |
| 22:50 |
<twentyafterfour@deploy1002> |
Started deploy [releng/phatality@27ddd0b]: deploy phatality |
[production] |
| 22:49 |
<twentyafterfour> |
deploying phatality |
[production] |
| 22:34 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2248.codfw.wmnet |
[production] |
| 22:33 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw2247.codfw.wmnet |
[production] |
| 22:17 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2247.codfw.wmnet |
[production] |
| 22:12 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2246.codfw.wmnet |
[production] |