4351-4400 of 10000 results (33ms)
2017-02-06 §
14:32 <ema> cp2018 stuck rebooting, powercycled [production]
14:30 <addshore@tin> Synchronized dblists/compact-language-links.dblist: T157108 & T157112 [[gerrit:335781|Deploy Compact Language Links out of beta in French/Dutch Wikipedia]] (duration: 00m 40s) [production]
14:23 <addshore@tin> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:333936|Copy InterwikiSorting settings from wmgWikibaseClientSettings]] (duration: 00m 40s) [production]
14:19 <marostegui> Stop MySQL and shutdown db2060 for maintenance - T156161 [production]
14:15 <addshore@tin> Synchronized wmf-config/InitialiseSettings.php: T155717 [[gerrit:332909|Enable TwoColConflict on mw.org]] (duration: 00m 40s) [production]
14:09 <addshore@tin> Synchronized php-1.29.0-wmf.10/extensions/ORES/extension.json: T157206 [[gerrit:336215|ORES - Remove all (except meta) API funcationality hooks]] (duration: 00m 51s) [production]
14:05 <volans> fixed duplicate entries in source.list on db2040 and es2002 (trusty) [production]
13:44 <ema> cache_misc: upgrade to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401 [production]
13:30 <elukey> applied https://gerrit.wikimedia.org/r/#/c/336203/ manually to analytics1028 (hadoop worker node) as live test - T156932 [production]
13:23 <gehel> removing stale puppet lock file on elastic10(22|26) [production]
12:03 <moritzm> upgrading mwdebug* and mw1261 to hhvm 3.12.11+dfsg-1+wmf2 [production]
10:21 <oblivian@tin> Finished deploy [changeprop/deploy@ac11ebe]: Deploying ores concurrency/disabling (duration: 00m 38s) [production]
10:20 <oblivian@tin> Started deploy [changeprop/deploy@ac11ebe]: Deploying ores concurrency/disabling [production]
10:18 <oblivian@tin> Started deploy [changeprop/deploy@ac11ebe]: Deploying ores concurrency/disabling to canary [production]
10:17 <gehel> data import complete for wdqs1003, repooling - T152643 [production]
10:14 <marostegui> Started to transfer commonswiki (ibd and cfg) from db1064 to labsdb1011 - T153743 [production]
10:14 <ema> cache_maps: upgrade to jessie 8.7 and reboot into kernel 4.4.2-3+wmf8 T155401 [production]
09:40 <gehel> elasticsearch - reindexing from 2017-02-04T20:00:00Z to 2017-02-05T23:59:00Z - T139043 [production]
09:36 <hoo> Removed 2fa from an account, per T157191 [production]
09:30 <marostegui> Stop MySQL Replication on db1064 for maintenance - T153743 [production]
09:26 <marostegui> Deploy ALTER table db1028 metawiki.pagelinks - T153300 [production]
09:25 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1028 - T153300 (duration: 00m 42s) [production]
08:43 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1064 - T153743 (duration: 00m 41s) [production]
08:19 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=aqs1008.eqiad.wmnet [production]
07:38 <elukey> bootstrapping aqs1009-a (new AQS cassandra instance) [production]
07:30 <marostegui> Stop MySQL on db1095 to snapshot it to es1017 - T153743 [production]
07:03 <marostegui> Upgrade mariadb+packages db1039 - T153300 [production]
07:01 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2060 - T156161 (duration: 00m 40s) [production]
02:25 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Feb 6 02:25:51 UTC 2017 (duration 5m 20s) [production]
02:20 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.10) (duration: 07m 30s) [production]
2017-02-05 §
23:42 <gehel> truncating elasticsearch logs on elastic10(24|26|40) - T139043 [production]
23:41 <gehel> truncating elasticsearch logs on elastic1022 - T139043 [production]
03:28 <Amir1> ladsgroup@scb100[1-4]:~$ sudo service celery-ores-worker restart (T157206) [production]
02:26 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Feb 5 02:26:26 UTC 2017 (duration 5m 18s) [production]
02:21 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.10) (duration: 08m 20s) [production]
2017-02-04 §
19:24 <elukey> Started nodetool-b cleanup on aqs1005 (after 1008-{ab} bootstraps) [production]
11:44 <elukey> Started nodetool-a cleanup on aqs1008 (after 1008-{ab} bootstraps) [production]
09:09 <elukey> Started nodetool-a cleanup on aqs1005 (after 1008-{ab} bootstraps) [production]
02:28 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Feb 4 02:28:33 UTC 2017 (duration 5m 23s) [production]
02:23 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.10) (duration: 07m 53s) [production]
2017-02-03 §
22:27 <mutante> switching webproxy.*.wmnet CNAMEs from carbon to new install servers (T123733) - watching squid access logs [production]
19:02 <ostriches> gerrit: flushed all web_sessions, you'll have to login again. Sorry [production]
18:07 <godog> stop carbon-cache on graphite1001 to prevent useless write load [production]
17:05 <godog> fail over read traffic from graphite1001 to graphite2001 https://gerrit.wikimedia.org/r/335761 - T157022 [production]
16:10 <godog> rsync coal data graphite1001 -> graphite2001 [production]
15:03 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2061 (duration: 00m 40s) [production]
15:01 <jynus> preparing to reimage db2039 T111654 [production]
14:35 <chasemp> restart apache on graphite1001 to see if it helps sqlite lock isssue [production]
14:30 <jynus> upgrade and restart db2061 T111654 [production]
14:26 <jynus@tin> Synchronized wmf-config/db-codfw.php: Depool db2061 (duration: 00m 40s) [production]