|
2018-02-13
§
|
| 11:41 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1099 (duration: 00m 56s) |
[production] |
| 11:37 |
<marostegui> |
Stop MySQL on db2059 and db2038 for kernel upgrade |
[production] |
| 11:29 |
<ema> |
lvs1003: restart pybal to reconnect to etcd |
[production] |
| 11:27 |
<ema> |
lvs1006/1010: restart pybal to reconnect to etcd |
[production] |
| 11:26 |
<ema> |
lvs4005: restart pybal to reconnect to etcd |
[production] |
| 11:23 |
<ema> |
esams primary LVSs: restart pybal to reconnect to etcd |
[production] |
| 11:21 |
<ema> |
esams secondary LVSs: restart pybal to properly reconnect to etcd |
[production] |
| 11:14 |
<ema> |
repool cp3007 |
[production] |
| 11:13 |
<ema> |
depool cp3007 to test pybal's behavior on lvs3002 |
[production] |
| 10:51 |
<filippo@tin> |
Synchronized wmf-config/ProductionServices.php: depool poolcounter1002 for disk replacement (duration: 00m 56s) |
[production] |
| 10:28 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1099 (duration: 00m 54s) |
[production] |
| 10:08 |
<godog> |
roll-restart ms-fe in codfw/eqiad after applying https://gerrit.wikimedia.org/r/c/409942/ |
[production] |
| 10:03 |
<ema> |
restart pybal on lvs2003 |
[production] |
| 09:58 |
<ema> |
restart pybal on lvs2006 |
[production] |
| 09:52 |
<filippo@neodymium> |
conftool action : set/pooled=no; selector: name=ms-fe2005.codfw.wmnet |
[production] |
| 09:49 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2075, depool db2038 and db2059 (duration: 00m 55s) |
[production] |
| 09:32 |
<marostegui> |
Stop mysql on db2075 for mysql and kernel upgrade |
[production] |
| 09:30 |
<marostegui> |
Stop replication in sync on db1089 and dbstore1002 - T162807 |
[production] |
| 09:29 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1065 - T162807 (duration: 00m 54s) |
[production] |
| 09:22 |
<elukey> |
powercycle analytics1062 - not reachable via ssh, frozen via serial console |
[production] |
| 09:22 |
<jynus> |
disabling puppet on all eqiad databases |
[production] |
| 09:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1065 - T162807 (duration: 00m 55s) |
[production] |
| 09:20 |
<marostegui> |
Stop replication in sync on db1089 and db1065 - T162807 |
[production] |
| 09:12 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2084:3315, depool db2075 (duration: 00m 55s) |
[production] |
| 09:01 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 - T162807 (duration: 00m 55s) |
[production] |
| 08:52 |
<marostegui> |
Stop replication in sync on db1089 and db1099:3311 - T162807 |
[production] |
| 08:51 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1089, db1099 - T162807 (duration: 00m 55s) |
[production] |
| 08:41 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2084:3315 (duration: 00m 56s) |
[production] |
| 08:37 |
<hashar> |
tin.eqiad.wmnet: removing live hack in /srv/mediawiki-staging/scap/plugins/clean.py | T187160 |
[production] |
| 08:32 |
<moritzm> |
installing wavpack security updates |
[production] |
| 08:09 |
<moritzm> |
installing exim security updates on trusty hosts |
[production] |
| 07:02 |
<marostegui> |
Deploy schema change on s5 db2089 db2084 db2075 db2039 db2059 - T187089 |
[production] |
| 06:28 |
<marostegui> |
reload haproxy on dbproxy1005 |
[production] |
| 05:10 |
<bblack@neodymium> |
conftool action : set/pooled=yes; selector: name=cp50(0[12345789]|1[12]).eqsin.wmnet |
[production] |
| 02:34 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.20) (duration: 05m 29s) |
[production] |
| 00:24 |
<cwd> |
re-enabled p-c |
[production] |
| 00:16 |
<demon@tin> |
Synchronized php-1.31.0-wmf.20/extensions/VisualEditor/modules/ve-mw/ui/pages/: T187112 (duration: 00m 56s) |
[production] |
| 00:10 |
<cwd> |
disabled p-c jobs for reboot |
[production] |
| 00:04 |
<demon@tin> |
Synchronized wmf-config/: cleanup Sentry inclusion for labs, should be no-op (duration: 00m 57s) |
[production] |
| 00:03 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: cleanup Sentry inclusion for labs, should be no-op (duration: 00m 56s) |
[production] |
|
2018-02-12
§
|
| 23:47 |
<demon@tin> |
Finished deploy [gerrit/gerrit@6adde70]: reviewers plugin (duration: 00m 12s) |
[production] |
| 23:46 |
<demon@tin> |
Started deploy [gerrit/gerrit@6adde70]: reviewers plugin |
[production] |
| 23:32 |
<mutante> |
terbium,wasat: touch /var/log/mediawwiki/purge_abusefilter.log ; set owner/permissions like other logfiles |
[production] |
| 23:13 |
<elukey> |
manual restart of Yarn Node Managers on analytics1058/31 (failed due to root partition filled up for the issue logged before) |
[production] |
| 23:09 |
<elukey> |
cleaned up tmp files on all analytics hadoop worker nodes, job filling up tmp |
[production] |
| 21:27 |
<andrew@tin> |
Finished deploy [horizon/deploy@2f70002]: updating several submodules, probably breaking static content (duration: 03m 18s) |
[production] |
| 21:24 |
<andrew@tin> |
Started deploy [horizon/deploy@2f70002]: updating several submodules, probably breaking static content |
[production] |
| 21:06 |
<mholloway-shell@tin> |
Finished deploy [mobileapps/deploy@0639c31]: Update mobileapps to f14bdd5 (duration: 05m 46s) |
[production] |
| 21:00 |
<mholloway-shell@tin> |
Started deploy [mobileapps/deploy@0639c31]: Update mobileapps to f14bdd5 |
[production] |
| 20:21 |
<andrew@tin> |
Finished deploy [horizon/deploy@c009388]: updating puppet dashboard (duration: 03m 22s) |
[production] |