3401-3450 of 10000 results (46ms)
2017-02-06 §
13:23 <gehel> removing stale puppet lock file on elastic10(22|26) [production]
12:03 <moritzm> upgrading mwdebug* and mw1261 to hhvm 3.12.11+dfsg-1+wmf2 [production]
10:21 <oblivian@tin> Finished deploy [changeprop/deploy@ac11ebe]: Deploying ores concurrency/disabling (duration: 00m 38s) [production]
10:20 <oblivian@tin> Started deploy [changeprop/deploy@ac11ebe]: Deploying ores concurrency/disabling [production]
10:18 <oblivian@tin> Started deploy [changeprop/deploy@ac11ebe]: Deploying ores concurrency/disabling to canary [production]
10:17 <gehel> data import complete for wdqs1003, repooling - T152643 [production]
10:14 <marostegui> Started to transfer commonswiki (ibd and cfg) from db1064 to labsdb1011 - T153743 [production]
10:14 <ema> cache_maps: upgrade to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401 [production]
09:40 <gehel> elasticsearch - reindexing from 2017-02-04T20:00:00Z to 2017-02-05T23:59:00Z - T139043 [production]
09:36 <hoo> Removed 2fa from an account, per T157191 [production]
09:30 <marostegui> Stop MySQL Replication on db1064 for maintenance - T153743 [production]
09:26 <marostegui> Deploy ALTER table db1028 metawiki.pagelinks - T153300 [production]
09:25 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1028 - T153300 (duration: 00m 42s) [production]
08:43 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1064 - T153743 (duration: 00m 41s) [production]
08:19 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=aqs1008.eqiad.wmnet [production]
07:38 <elukey> bootstrapping aqs1009-a (new AQS cassandra instance) [production]
07:30 <marostegui> Stop MySQL on db1095 to snapshot it to es1017 - T153743 [production]
07:03 <marostegui> Upgrade mariadb+packages db1039 - T153300 [production]
07:01 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2060 - T156161 (duration: 00m 40s) [production]
02:25 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Feb 6 02:25:51 UTC 2017 (duration 5m 20s) [production]
02:20 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.10) (duration: 07m 30s) [production]
2017-02-05 §
23:42 <gehel> truncating elasticsearch logs on elastic10(24|26|40) - T139043 [production]
23:41 <gehel> truncating elasticsearch logs on elastic1022 - T139043 [production]
03:28 <Amir1> ladsgroup@scb100[1-4]:~$ sudo service celery-ores-worker restart (T157206) [production]
02:26 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Feb 5 02:26:26 UTC 2017 (duration 5m 18s) [production]
02:21 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.10) (duration: 08m 20s) [production]
2017-02-04 §
19:24 <elukey> Started nodetool-b cleanup on aqs1005 (after 1008-{ab} bootstraps) [production]
11:44 <elukey> Started nodetool-a cleanup on aqs1008 (after 1008-{ab} bootstraps) [production]
09:09 <elukey> Started nodetool-a cleanup on aqs1005 (after 1008-{ab} bootstraps) [production]
02:28 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Feb 4 02:28:33 UTC 2017 (duration 5m 23s) [production]
02:23 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.10) (duration: 07m 53s) [production]
2017-02-03 §
22:27 <mutante> switching webproxy.*.wmnet CNAMEs from carbon to new install servers (T123733) - watching squid access logs [production]
19:02 <ostriches> gerrit: flushed all web_sessions, you'll have to login again. Sorry [production]
18:07 <godog> stop carbon-cache on graphite1001 to prevent useless write load [production]
17:05 <godog> fail over read traffic from graphite1001 to graphite2001 https://gerrit.wikimedia.org/r/335761 - T157022 [production]
16:10 <godog> rsync coal data graphite1001 -> graphite2001 [production]
15:03 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2061 (duration: 00m 40s) [production]
15:01 <jynus> preparing to reimage db2039 T111654 [production]
14:35 <chasemp> restart apache on graphite1001 to see if it helps sqlite lock isssue [production]
14:30 <jynus> upgrade and restart db2061 T111654 [production]
14:26 <jynus@tin> Synchronized wmf-config/db-codfw.php: Depool db2061 (duration: 00m 40s) [production]
13:58 <jynus> restarting and upgrading db2041 T111654 [production]
12:16 <gehel> restarting relforge1001 to pick up new master configuration [production]
11:21 <jynus> preparing to reimage db2054 T111654 [production]
11:01 <marostegui> Alter table metawiki.pagelinks on db1039 (depooled) - T153300 [production]
10:54 <jynus> preparing to reimage db2053 T111654 [production]
10:50 <moritzm> mwdebug* and mw1261 have been reverted to previous HHVM package [production]
10:19 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1064 - T153743 (duration: 00m 42s) [production]
10:04 <marostegui> Reboot db1064 to pick up the new kernels T153743 [production]
09:59 <marostegui> Upgrade db1064 from MariaDB 10.0.23 to 10.0.29 - T153743 [production]