|
2020-04-27
ยง
|
| 14:27 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 14:27 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 14:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1078 for schema change', diff saved to https://phabricator.wikimedia.org/P11045 and previous config saved to /var/cache/conftool/dbconfig/20200427-142006-marostegui.json |
[production] |
| 13:56 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 13:55 |
<vgutierrez> |
depool cp4026 and upgrade to ATS 8.1.0 - T249335 |
[production] |
| 13:53 |
<vgutierrez> |
restart ats-tls on cp3056 - T249335 |
[production] |
| 13:52 |
<mutante> |
decom'ing install1002 and install2002 - see install1003/2003 and apt1001/2001 |
[production] |
| 13:52 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 13:51 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 13:47 |
<marostegui> |
Deploy schema change on s3 codfw, lag will show up - T250055 |
[production] |
| 13:46 |
<marostegui> |
Drop img_deleted column from wikitech - T250055 |
[production] |
| 13:45 |
<marostegui> |
Drop img_deleted column from s7 eqiad - T250055 |
[production] |
| 13:42 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 13:42 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-analytics' for release 'canary' . |
[production] |
| 13:42 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 13:41 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 13:41 |
<dzahn@cumin1001> |
END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) |
[production] |
| 13:41 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 13:38 |
<_joe_> |
repooling both mw1407 and mw1409 for tesing T99740 |
[production] |
| 13:30 |
<_joe_> |
depooled mw1409 as well as mw1407 for further benchmarking, T99740 |
[production] |
| 13:28 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-analytics' for release 'canary' . |
[production] |
| 13:28 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 13:10 |
<elukey> |
roll restart elastic on cloudelastic-chi again to pick up new JVM settings - T231517 |
[production] |
| 13:09 |
<marostegui> |
Deploy schema change on s7 codfw, lag will show up - T250055 |
[production] |
| 12:53 |
<marostegui> |
Drop T248086_wb_terms from db1104 - T248086 |
[production] |
| 12:50 |
<marostegui> |
Removed img_deleted from s1 (enwiki) T250055 |
[production] |
| 12:49 |
<akosiaris> |
rolling back etherpad to 1.8.0 |
[production] |
| 12:45 |
<akosiaris> |
upgrade etherpad to 1.8.3 |
[production] |
| 12:41 |
<marostegui> |
Remove empty table T248086_wb_terms from wikidatawiki on s3 eqiad - T248086 |
[production] |
| 12:36 |
<marostegui> |
Remove empty table T248086_wb_terms from wikidatawiki on s3 codfw master - T248086 |
[production] |
| 12:32 |
<marostegui> |
Remove empty table T248086_wb_terms from wikidatawiki on s8 codfw master - T248086 |
[production] |
| 12:14 |
<marostegui> |
Remove empty table T248086_wb_terms from commonswiki and testcommonswiki on s4 master - T248086 |
[production] |
| 12:06 |
<ema> |
cp: upgrade purged to 0.8 T249583 |
[production] |
| 11:53 |
<Lucas_WMDE> |
EU SWAT done |
[production] |
| 11:49 |
<hoo> |
Started Wikibase rebuildItemsPerSite on mwmaint1002 for wikidatawiki. Can be killed at any time, if necessary. (T249613) |
[production] |
| 11:46 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:592515|Enable cross-project search on frwiktionary (T250724)]] (duration: 00m 57s) |
[production] |
| 11:41 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:592330|Add transwiki import sources in zhwiki (T250972)]] (duration: 00m 57s) |
[production] |
| 11:25 |
<addshore> |
repool wdqs1007 T242453 |
[production] |
| 11:21 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:592306|Add two domains in wgCopyUploadsDomains (T250903, T250904)]] (duration: 00m 57s) |
[production] |
| 11:20 |
<_joe_> |
restarted php-fpm on mw1407 to pick up enlarged opcache values, T99740 |
[production] |
| 11:14 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:592634| Bumping portals to master (563985)]] (duration: 00m 57s) |
[production] |
| 11:13 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:592634| Bumping portals to master (563985)]] (duration: 00m 58s) |
[production] |
| 11:09 |
<cparle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [SDC] Enable constraints on production commons (duration: 00m 57s) |
[production] |
| 11:08 |
<cparle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [SDC] Enable constraints on production commons (duration: 00m 58s) |
[production] |
| 10:52 |
<hoo> |
Running the pruneItemsPerSite on mwmaint1002 maintenance script for Wikidata (T249613) |
[production] |
| 10:52 |
<hoo@deploy1001> |
Synchronized php-1.35.0-wmf.28/extensions/Wikibase: pruneItemsPerSite: Fix join_condition call signature (T249613) (duration: 01m 02s) |
[production] |
| 10:49 |
<hoo@deploy1001> |
Synchronized php-1.35.0-wmf.28/extensions/Wikibase: pruneItemsPerSite: Fix join_condition call signature (T249613) (duration: 01m 01s) |
[production] |
| 10:32 |
<mutante> |
contint2001 - systemd status was degraded. icinga alerted. failed unit was jenkins. starting it failed with "address already in use". manually started without using systemctl? killed jenkins and started again with systemctl. T224591 |
[production] |
| 10:29 |
<mutante> |
contint2001 - jenkins failed and can't start because address is already in use |
[production] |
| 10:22 |
<addshore> |
depool and restart wdqs1007 (deadlocks) T242453 |
[production] |