|
2016-06-27
§
|
| 15:16 |
<gehel> |
banning elastic1001 to prepare its decommissioning (T138329) |
[production] |
| 15:13 |
<thcipriani@tin> |
Synchronized dblists/clldefault.dblist: SWAT: [[gerrit:296180|Deploy Compact Language Links as default (Stage 3)]] PART II (duration: 00m 23s) |
[production] |
| 15:07 |
<thcipriani@tin> |
Synchronized wmf-config: SWAT: [[gerrit:296180|Deploy Compact Language Links as default (Stage 3)]] (duration: 00m 40s) |
[production] |
| 15:00 |
<elukey> |
mw1136 powercycled - not responsive to ssh and root login |
[production] |
| 14:49 |
<gehel@palladium> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic101[0-6].eqiad.wmnet |
[production] |
| 14:39 |
<gehel@palladium> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic100[0-9].eqiad.wmnet |
[production] |
| 14:37 |
<gehel@palladium> |
conftool action : get/pooled; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic100[0-9]..eqiad.wmnet |
[production] |
| 14:34 |
<gehel> |
removing old elasticsearch servers in eqiad from LVS (elastic1001-1016 - T138329) |
[production] |
| 10:09 |
<moritzm> |
pooled mw1291 (jessie imagescaler) |
[production] |
| 09:48 |
<jynus> |
stopping and reimporting db2010 (m1) |
[production] |
| 09:47 |
<gehel> |
removing maps-test*.codfw.wmnet servers from LVS (T138092) |
[production] |
| 09:19 |
<gehel@palladium> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch-ssl,name=elastic104..eqiad.wmnet |
[production] |
| 09:19 |
<gehel@palladium> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic104..eqiad.wmnet |
[production] |
| 09:18 |
<gehel@palladium> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch-ssl,name=elastic103..eqiad.wmnet |
[production] |
| 09:18 |
<gehel@palladium> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic103..eqiad.wmnet |
[production] |
| 09:10 |
<gehel@palladium> |
conftool action : get/pooled; selector: elastic10??\.eqiad\.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch']) |
[production] |
| 09:07 |
<gehel@palladium> |
conftool action : set/pooled=yes; selector: elastic1032.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch-ssl']) |
[production] |
| 09:06 |
<gehel@palladium> |
conftool action : set/pooled=yes; selector: elastic1032.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch']) |
[production] |
| 09:00 |
<gehel> |
adding new elasticsearch servers in eqiad to LVS |
[production] |
| 08:54 |
<godog> |
swift codfw-prod ms-be202[234] weight 2000 |
[production] |
| 07:15 |
<elukey> |
puppet stopped on analytics1049 to remove it completely from the Hadoop cluster - broken disk |
[production] |
| 02:51 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Jun 27 02:51:41 UTC 2016 (duration 7m 5s) |
[production] |
| 02:44 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.7) (duration: 08m 09s) |
[production] |
| 02:27 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 54s) |
[production] |
|
2016-06-24
§
|
| 18:41 |
<krenair@tin> |
Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295958/4 - fix mobile main page rendering on a bunch of wikis, effectively putting them back to how they were a few days ago (duration: 00m 37s) |
[production] |
| 17:19 |
<mobrovac> |
change-prop deploying df88a75b |
[production] |
| 17:05 |
<_joe_> |
re-started changeprop after disabling the dependency module |
[production] |
| 14:18 |
<paravoid> |
shutting down ms-fe3002 due to on-site work |
[production] |
| 14:05 |
<krinkle@tin> |
Synchronized php-1.28.0-wmf.7/includes/OutputPage.php: T138586 hotfix (duration: 00m 47s) |
[production] |
| 14:02 |
<mobrovac> |
scb100x disabled puppet to clear changeprop queues |
[production] |
| 13:22 |
<gehel> |
re-enabling puppet on maps1002 (still in pre-configuration state, only default role) |
[production] |
| 12:34 |
<hashar> |
Random resource loader entries are apparently faulty causing issues with css and/or javascript T138586 |
[production] |
| 12:04 |
<elukey@palladium> |
conftool action : set/pooled=no; selector: aqs1001.eqiad.wmnet |
[production] |
| 12:03 |
<elukey> |
rebooting aqs1001.eqiad.wmnet for kernel upgrades |
[production] |
| 10:55 |
<jynus> |
updated m1-slave dns to be db1001 |
[production] |
| 10:20 |
<hashar> |
gallium: restarted apache2 , potentially stuck proxy |
[production] |
| 10:18 |
<moritzm> |
upgrade nodejs on scb systems in codfw and restart node-based services |
[production] |
| 09:59 |
<ema> |
nginx rolling restart to enable TFO on all tlsproxies (T108827) |
[production] |
| 09:52 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1059 with low weight, increase weight of db1061, db1062 (duration: 00m 33s) |
[production] |
| 09:48 |
<moritzm> |
upgrade nodejs on restbase test systems (xenon/praseodymium/cerium/restbase-test) and restart restbase on those |
[production] |
| 09:09 |
<mobrovac> |
scb100x stopping puppet to stop change-prop and clear the queue |
[production] |