1101-1150 of 10000 results (47ms)
2017-07-17 §
14:34 <marostegui> Run maintain-views on labsdb1009,10 and 11 for s6 - T153743 [production]
14:13 <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:365092|Add HD logos for several Wiktionaries (T150618)]] (duration: 00m 46s) [production]
14:11 <zfilipin@tin> Synchronized static/images/project-logos/: SWAT: [[gerrit:365092|Add HD logos for several Wiktionaries (T150618)]] (duration: 00m 49s) [production]
14:11 <ottomata> decommissioning rcs100[12] to spare::system: T170157 [production]
14:02 <zeljkof> Extending EU SWAT [production]
13:59 <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:365086|Provide HD logos for several Wikiversities (T150618)]] (duration: 00m 46s) [production]
13:58 <zfilipin@tin> Synchronized static/images/project-logos: SWAT: [[gerrit:365086|Provide HD logos for several Wikiversities (T150618)]] (duration: 00m 47s) [production]
13:46 <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:365084|Provide HD logos for several Wikipedias (T150618)]] (duration: 00m 46s) [production]
13:43 <marostegui> Deploy alter table on s4 - dbstore1001 - T168661 [production]
13:37 <zfilipin@tin> scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) [production]
13:36 <zfilipin@tin> Synchronized static/images/project-logos/: SWAT: [[gerrit:365084|Provide HD logos for several Wikipedias (T150618)]] (duration: 00m 48s) [production]
11:35 <marostegui> Deploy alter table on s4 - dbstore1002 - T168661 [production]
11:33 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1056 - T168661 (duration: 00m 46s) [production]
10:16 <moritzm> installing apache updates on graphite hosts [production]
10:02 <moritzm> installing apache updates on logstash [production]
10:01 <moritzm> installing apache updates on otrs.wikimedia.org [production]
09:48 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2062 (duration: 00m 47s) [production]
09:24 <marostegui> Disable puppet on labsdb1010 for maintenance - T153743 [production]
09:22 <marostegui> Stop replication on labsdb1009 and labsdb1010 for maintenance - T153743 [production]
09:05 <marostegui> Disable puppet on labsdb1009 for maintenance - T153743 [production]
08:28 <akosiaris> reboot helium/heze for kernel upgrades [production]
08:23 <marostegui> Deploy alter table s1 - labsdb1001 - T166204 [production]
08:20 <marostegui> Increase expire_logs_days on db1069:3311 from 7 to 14 temporarily - T166204 [production]
08:17 <ema> lvs100[39]: upgrade pybal to 1.13.9 T82747 T154759 [production]
08:06 <ema> lvs2003: upgrade pybal to 1.13.9 T82747 T154759 [production]
07:57 <ema@neodymium> conftool action : set/pooled=inactive; selector: name=wdqs1002.eqiad.wmnet [production]
07:55 <akosiaris> upgrade nodejs to 6.11 on etherpad1001 [production]
07:32 <moritzm> updating ruthenium to nodejs 6.11 [production]
07:12 <marostegui> Stop slave s2 on db1102 for maintenance - T153743 [production]
07:09 <marostegui> Deploy alter table s4 - db1056 - T168661 [production]
07:06 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 46s) [production]
07:05 <marostegui@tin> scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) [production]
07:00 <marostegui> Rename labsdb1011 main replication thread to an specific one - T153743 [production]
06:50 <marostegui> Stop replication on db1095 for maintenance - T153743 [production]
06:48 <marostegui> Deploy alter table on s1 - db1073 - T166204 [production]
06:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1073 - T166204 (duration: 01m 04s) [production]
05:21 <marostegui> Add 50G to /srv on db1069 [production]
05:09 <marostegui> Restart MySQL on labsdb1009 for maintenance - T170657 [production]
03:13 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Jul 17 03:13:51 UTC 2017 (duration 7m 16s) [production]
03:06 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.9) (duration: 12m 48s) [production]
02:31 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.7) (duration: 09m 13s) [production]
2017-07-15 §
10:42 <elukey> puppetdb restarted on nitrogen, puppet agents re-enabled - T170740 [production]
10:06 <akosiaris> disable puppet on the entire fleet for puppetdb debugging on nitrogen [production]
02:53 <mutante> servermon - switched to netmon1003 backend (jessie ganeti) [production]
02:46 <mutante> netmon1001 - stopping "make_updates" cron , migration to netmon1003, flipping cache::backend to netmon1003 (T170653) [production]
2017-07-14 §
21:48 <mutante> netmon1003 - reinstalled with jessie - saw nothing on ganeti console at all which was a bit confusing, but install finished anyways - adding to puppet / signing cert (T170655) [production]
20:47 <bblack> mailbox lag: restarting cp1074 backend [production]
19:50 <mutante> wikitech-static: re-enabled HSTS - line was commented out in Apache config, activated it again [production]
18:54 <herron> added exim from/subject filter for spam observed from qq.com - T170601 [production]
16:36 <herron> lowered mailman/lists spam_score exim acl to 6 - T170601 [production]