1151-1200 of 10000 results (50ms)
2018-02-13 §
11:57 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2066 (duration: 00m 55s) [production]
11:56 <marostegui> Deploy schema change on db2066 - T187089 T185128 T153182 [production]
11:50 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Rpool db2038 and db2059 (duration: 00m 55s) [production]
11:47 <jynus> reenabling puppet on all eqiad databases [production]
11:41 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1099 (duration: 00m 56s) [production]
11:37 <marostegui> Stop MySQL on db2059 and db2038 for kernel upgrade [production]
11:29 <ema> lvs1003: restart pybal to reconnect to etcd [production]
11:27 <ema> lvs1006/1010: restart pybal to reconnect to etcd [production]
11:26 <ema> lvs4005: restart pybal to reconnect to etcd [production]
11:23 <ema> esams primary LVSs: restart pybal to reconnect to etcd [production]
11:21 <ema> esams secondary LVSs: restart pybal to properly reconnect to etcd [production]
11:14 <ema> repool cp3007 [production]
11:13 <ema> depool cp3007 to test pybal's behavior on lvs3002 [production]
10:51 <filippo@tin> Synchronized wmf-config/ProductionServices.php: depool poolcounter1002 for disk replacement (duration: 00m 56s) [production]
10:28 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool db1099 (duration: 00m 54s) [production]
10:08 <godog> roll-restart ms-fe in codfw/eqiad after applying https://gerrit.wikimedia.org/r/c/409942/ [production]
10:03 <ema> restart pybal on lvs2003 [production]
09:58 <ema> restart pybal on lvs2006 [production]
09:52 <filippo@neodymium> conftool action : set/pooled=no; selector: name=ms-fe2005.codfw.wmnet [production]
09:49 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2075, depool db2038 and db2059 (duration: 00m 55s) [production]
09:32 <marostegui> Stop mysql on db2075 for mysql and kernel upgrade [production]
09:30 <marostegui> Stop replication in sync on db1089 and dbstore1002 - T162807 [production]
09:29 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1065 - T162807 (duration: 00m 54s) [production]
09:22 <elukey> powercycle analytics1062 - not reachable via ssh, frozen via serial console [production]
09:22 <jynus> disabling puppet on all eqiad databases [production]
09:20 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1065 - T162807 (duration: 00m 55s) [production]
09:20 <marostegui> Stop replication in sync on db1089 and db1065 - T162807 [production]
09:12 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2084:3315, depool db2075 (duration: 00m 55s) [production]
09:01 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 - T162807 (duration: 00m 55s) [production]
08:52 <marostegui> Stop replication in sync on db1089 and db1099:3311 - T162807 [production]
08:51 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1089, db1099 - T162807 (duration: 00m 55s) [production]
08:41 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2084:3315 (duration: 00m 56s) [production]
08:37 <hashar> tin.eqiad.wmnet: removing live hack in /srv/mediawiki-staging/scap/plugins/clean.py | T187160 [production]
08:32 <moritzm> installing wavpack security updates [production]
08:09 <moritzm> installing exim security updates on trusty hosts [production]
07:02 <marostegui> Deploy schema change on s5 db2089 db2084 db2075 db2039 db2059 - T187089 [production]
06:28 <marostegui> reload haproxy on dbproxy1005 [production]
05:10 <bblack@neodymium> conftool action : set/pooled=yes; selector: name=cp50(0[12345789]|1[12]).eqsin.wmnet [production]
02:34 <l10nupdate@tin> scap sync-l10n completed (1.31.0-wmf.20) (duration: 05m 29s) [production]
00:24 <cwd> re-enabled p-c [production]
00:16 <demon@tin> Synchronized php-1.31.0-wmf.20/extensions/VisualEditor/modules/ve-mw/ui/pages/: T187112 (duration: 00m 56s) [production]
00:10 <cwd> disabled p-c jobs for reboot [production]
00:04 <demon@tin> Synchronized wmf-config/: cleanup Sentry inclusion for labs, should be no-op (duration: 00m 57s) [production]
00:03 <demon@tin> Synchronized wmf-config/InitialiseSettings.php: cleanup Sentry inclusion for labs, should be no-op (duration: 00m 56s) [production]
2018-02-12 §
23:47 <demon@tin> Finished deploy [gerrit/gerrit@6adde70]: reviewers plugin (duration: 00m 12s) [production]
23:46 <demon@tin> Started deploy [gerrit/gerrit@6adde70]: reviewers plugin [production]
23:32 <mutante> terbium,wasat: touch /var/log/mediawwiki/purge_abusefilter.log ; set owner/permissions like other logfiles [production]
23:13 <elukey> manual restart of Yarn Node Managers on analytics1058/31 (failed due to root partition filled up for the issue logged before) [production]
23:09 <elukey> cleaned up tmp files on all analytics hadoop worker nodes, job filling up tmp [production]
21:27 <andrew@tin> Finished deploy [horizon/deploy@2f70002]: updating several submodules, probably breaking static content (duration: 03m 18s) [production]