|
2020-04-13
§
|
| 11:53 |
<marostegui> |
Deploy schema change on codfw master (lag will appear on codfw) - T250062 |
[production] |
| 11:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: efe2feb: robots.txt: Disable indexing user (sub)pages and draft-related pages on srwiki (T248860; take II) (duration: 00m 58s) |
[production] |
| 11:14 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: efe2feb: robots.txt: Disable indexing user (sub)pages and draft-related pages on srwiki (T248860) (duration: 00m 58s) |
[production] |
| 10:37 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:588383| Bumping portals to master (563985)]] (duration: 00m 58s) |
[production] |
| 10:36 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:588383| Bumping portals to master (563985)]] (duration: 01m 00s) |
[production] |
| 10:24 |
<mutante> |
depooled wdqs1004 by request because of high lag |
[production] |
| 10:19 |
<marostegui> |
Kill updateSpecialPages.php --only=Fewestrevisions for s8 in mwmaint1002, the vslow host is lagging and creating errors |
[production] |
| 10:12 |
<mutante> |
mwmaint1002 - sudo systemctl status mediawiki_job_translationnotifications-mediawikiwiki.service |
[production] |
| 10:00 |
<mutante> |
- phabricator-stage-1001: replace deployment-tin.deployment-rep with deploy-1002.devtools in deployment-cache/.config |
[devtools] |
| 09:52 |
<Urbanecm> |
Rename user account Gerakiw@grwikimedia to Geraki@grwikimedia (T245911) |
[production] |
| 09:47 |
<Urbanecm> |
mwscript createAndPromote.php --wiki=grwikimedia --force Gerakiw <redacted> (T245911) |
[production] |
| 09:40 |
<mutante> |
set missing (and new) profile::tlsproxy::envoy::capitalize_headers: true to fix puppet errors |
[devtools] |
| 09:35 |
<mutante> |
set phabricator::vcs::address::v6 to fe80 local address to fix puppet error on phabricator-stage-1001 |
[devtools] |
| 08:15 |
<marostegui> |
Remove grants for haproxy@10.64.37.15 from labsdb hosts T231280 |
[production] |
| 07:50 |
<vgutierrez> |
enable memory tracking in ats-tls on cp1085 - T249335 |
[production] |
| 07:43 |
<marostegui> |
Compress db1092 T232446 |
[production] |
| 07:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Temporary pool db1111 in s8 API', diff saved to https://phabricator.wikimedia.org/P10964 and previous config saved to /var/cache/conftool/dbconfig/20200413-074158-marostegui.json |
[production] |
| 07:40 |
<vgutierrez> |
rolling upgrade to ats 8.0.7-rc0-1wm1 in ulsfo |
[production] |
| 07:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092 T232446', diff saved to https://phabricator.wikimedia.org/P10963 and previous config saved to /var/cache/conftool/dbconfig/20200413-073939-marostegui.json |
[production] |
| 07:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1110 T249973', diff saved to https://phabricator.wikimedia.org/P10962 and previous config saved to /var/cache/conftool/dbconfig/20200413-071740-marostegui.json |
[production] |
| 06:51 |
<marostegui> |
Deploy schema changes on db1110 - T249973 |
[production] |
| 06:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1110 T249973', diff saved to https://phabricator.wikimedia.org/P10961 and previous config saved to /var/cache/conftool/dbconfig/20200413-065022-marostegui.json |
[production] |
| 06:36 |
<elukey> |
temporary stopped puppet on restbase2014 to avoid attempts to start cassandra on each run - T250050 |
[production] |
| 06:23 |
<vgutierrez> |
upgrade to ats 8.0.7-rc0-1wm1 on cp[4026,4032,5006,5012] |
[production] |
| 06:20 |
<vgutierrez> |
upload trafficserver 8.0.7-rc0-1wm1 to apt.wm.o (buster) |
[production] |
| 05:25 |
<vgutierrez> |
restart varnish-fe on cp3050 |
[production] |
|
2020-04-12
§
|
| 11:11 |
<vgutierrez> |
restart ats-tls on cp5008.eqsin.wmnet - T249335 |
[production] |
| 10:18 |
<elukey> |
restart wdqs-updater on wdqs1004 (logs show no reports from the past hours, last one were stack traces related to a json decode failure) |
[production] |
| 06:59 |
<dcausse> |
restarting blazegraph on wdqs1004 (T242453) |
[production] |
| 06:35 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1025.eqiad.wmnet |
[production] |
| 06:32 |
<elukey> |
powerdown restbase1025 - T250027 |
[production] |
| 06:20 |
<elukey> |
powercycle restbase1025 (not reachable, serial console shows blank, racadm getsel reports errors with DIMM_B2) |
[production] |
| 05:53 |
<bblack> |
pushing https://gerrit.wikimedia.org/r/588134 to cache_text |
[production] |
| 05:50 |
<vgutierrez> |
restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335 |
[production] |
| 05:31 |
<bblack> |
pushing https://gerrit.wikimedia.org/r/588133 to cache_text |
[production] |
| 04:11 |
<bd808> |
Hopefully fixed T243843, T243843, T243843, and T243843 (deliberate duplication there folks) |
[tools.stashbot] |
| 04:09 |
<bd808> |
--canonical for webservice |
[tools.stashbot] |
| 03:59 |
<bd808> |
test |
[tools.stashbot] |
| 02:58 |
<bd808> |
Set --canonical to force redirect to sal.toolforge.org and added service.template to make this all easier in the future |
[tools.sal] |
| 02:52 |
<wm-bot> |
<bd808> Updated to 49015bb: Manually setup Elasticsearch creds (T247715) |
[tools.sal] |
| 00:11 |
<bd808> |
Everything broken at the moment because of elasticsearch7 migration not going as hoped. |
[tools.sal] |
|
2020-04-11
§
|
| 23:17 |
<wm-bot> |
<bd808> Updated config to point to es7 cluster (T247715) |
[tools.sal] |
| 23:11 |
<wm-bot> |
<bd808> Updated to f2ca4e4 925b463 Update !log handling for es7 (T247715) |
[tools.stashbot] |
| 22:59 |
<wm-bot> |
<bd808> Updated to f2ca4e4 Update !bash handling for es7 (T247715) |
[tools.stashbot] |
| 19:52 |
<cdanis@cumin1001> |
dbctl commit (dc=all): 'slight deweight to db1111', diff saved to https://phabricator.wikimedia.org/P10960 and previous config saved to /var/cache/conftool/dbconfig/20200411-195235-cdanis.json |
[production] |
| 17:35 |
<cdanis@cumin1001> |
dbctl commit (dc=all): 's8: +weight db1111, -weight db1126', diff saved to https://phabricator.wikimedia.org/P10959 and previous config saved to /var/cache/conftool/dbconfig/20200411-173517-cdanis.json |
[production] |
| 15:39 |
<vgutierrez> |
restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335 |
[production] |
| 15:07 |
<Krenair> |
Migrated from deployment-cache-text05 (stretch) to deployment-cache-text06 (buster) - class stopped working on stretch with https://gerrit.wikimedia.org/r/c/operations/puppet/+/584553 - shut down old instance - T250006 |
[releng] |
| 14:52 |
<Krenair> |
Migrated from deployment-cache-upload05 (stretch) to deployment-cache-upload06 (buster) - class stopped working on stretch with https://gerrit.wikimedia.org/r/c/operations/puppet/+/584553 - shut down old instance which coincidentally would turn one year old tomorrow |
[releng] |
| 09:30 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) |
[production] |