401-450 of 10000 results (37ms)
2017-02-06 §
09:36 <hoo> Removed 2fa from an account, per T157191 [production]
09:30 <marostegui> Stop MySQL Replication on db1064 for maintenance - T153743 [production]
09:26 <marostegui> Deploy ALTER table db1028 metawiki.pagelinks - T153300 [production]
09:25 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1028 - T153300 (duration: 00m 42s) [production]
08:43 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1064 - T153743 (duration: 00m 41s) [production]
08:19 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=aqs1008.eqiad.wmnet [production]
07:38 <elukey> bootstrapping aqs1009-a (new AQS cassandra instance) [production]
07:30 <marostegui> Stop MySQL on db1095 to snapshot it to es1017 - T153743 [production]
07:03 <marostegui> Upgrade mariadb+packages db1039 - T153300 [production]
07:01 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2060 - T156161 (duration: 00m 40s) [production]
02:25 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Feb 6 02:25:51 UTC 2017 (duration 5m 20s) [production]
02:20 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.10) (duration: 07m 30s) [production]
2017-02-05 §
23:42 <gehel> truncating elasticsearch logs on elastic10(24|26|40) - T139043 [production]
23:41 <gehel> truncating elasticsearch logs on elastic1022 - T139043 [production]
03:28 <Amir1> ladsgroup@scb100[1-4]:~$ sudo service celery-ores-worker restart (T157206) [production]
02:26 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Feb 5 02:26:26 UTC 2017 (duration 5m 18s) [production]
02:21 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.10) (duration: 08m 20s) [production]
2017-02-04 §
19:24 <elukey> Started nodetool-b cleanup on aqs1005 (after 1008-{ab} bootstraps) [production]
11:44 <elukey> Started nodetool-a cleanup on aqs1008 (after 1008-{ab} bootstraps) [production]
09:09 <elukey> Started nodetool-a cleanup on aqs1005 (after 1008-{ab} bootstraps) [production]
02:28 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Feb 4 02:28:33 UTC 2017 (duration 5m 23s) [production]
02:23 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.10) (duration: 07m 53s) [production]
2017-02-03 §
22:27 <mutante> switching webproxy.*.wmnet CNAMEs from carbon to new install servers (T123733) - watching squid access logs [production]
19:02 <ostriches> gerrit: flushed all web_sessions, you'll have to login again. Sorry [production]
18:07 <godog> stop carbon-cache on graphite1001 to prevent useless write load [production]
17:05 <godog> fail over read traffic from graphite1001 to graphite2001 https://gerrit.wikimedia.org/r/335761 - T157022 [production]
16:10 <godog> rsync coal data graphite1001 -> graphite2001 [production]
15:03 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2061 (duration: 00m 40s) [production]
15:01 <jynus> preparing to reimage db2039 T111654 [production]
14:35 <chasemp> restart apache on graphite1001 to see if it helps sqlite lock isssue [production]
14:30 <jynus> upgrade and restart db2061 T111654 [production]
14:26 <jynus@tin> Synchronized wmf-config/db-codfw.php: Depool db2061 (duration: 00m 40s) [production]
13:58 <jynus> restarting and upgrading db2041 T111654 [production]
12:16 <gehel> restarting relforge1001 to pick up new master configuration [production]
11:21 <jynus> preparing to reimage db2054 T111654 [production]
11:01 <marostegui> Alter table metawiki.pagelinks on db1039 (depooled) - T153300 [production]
10:54 <jynus> preparing to reimage db2053 T111654 [production]
10:50 <moritzm> mwdebug* and mw1261 have been reverted to previous HHVM package [production]
10:19 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1064 - T153743 (duration: 00m 42s) [production]
10:04 <marostegui> Reboot db1064 to pick up the new kernels T153743 [production]
09:59 <marostegui> Upgrade db1064 from MariaDB 10.0.23 to 10.0.29 - T153743 [production]
09:55 <gehel> restarting relforge1002 to pick up new master configuration [production]
09:54 <jynus> upgrade & restart of db2063 T111654 [production]
09:48 <marostegui> Restart mysql on db1064 to get its binary log changed to ROW - T153743 [production]
09:42 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1064 - T153743 (duration: 00m 40s) [production]
09:41 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db1064 - T153743 (duration: 00m 40s) [production]
09:39 <moritzm> upgraded mwdebug* and mw1261 to the new HHVM package [production]
09:22 <moritzm> uploaded hhvm_3.12.11+dfsg-1+wmf2 to apt.wikimedia.org [production]
09:10 <elukey> Replace Redis/Memcached shards mc2008->2011 with mc2026->mc2029 [production]
08:50 <moritzm> installing tomcat regression updates on trusty hosts (jessie update was fine) [production]