551-600 of 10000 results (19ms)
2014-02-18 §
02:48 <LocalisationUpdate> ResourceLoader cache refresh completed at 2014-02-18 02:48:14+00:00 [production]
02:42 <awight> rollback payments from 9e4d8b29581e2465d1acde8d7c2377fa6a8522a6 to 9b936320b797bd01c4e61b1cd7c2e15b0820a24b [production]
02:38 <awight> rollback payments from ce6233998f4bc0266c2e027c44620a8ba9984681 to 9e4d8b29581e2465d1acde8d7c2377fa6a8522a6 [production]
02:26 <LocalisationUpdate> completed (1.23wmf14) at 2014-02-18 02:26:05+00:00 [production]
02:14 <LocalisationUpdate> completed (1.23wmf13) at 2014-02-18 02:14:08+00:00 [production]
2014-02-17 §
20:57 <ottomata> depooling cp3022.esams.wikimedia.org to investigate varnishkafka issues [production]
16:15 <hashar> Jenkins deleting slave integration-slave01 (had only 2 CPU) [production]
16:14 <hashar> Jenkins added two labs slaves with 4 CPU: integration-slave02 and integration-slave03 [production]
08:46 <hashar> Upgrading Jenkins, half an hour downtime [production]
03:19 <LocalisationUpdate> ResourceLoader cache refresh completed at 2014-02-17 03:19:28+00:00 [production]
02:37 <LocalisationUpdate> completed (1.23wmf14) at 2014-02-17 02:37:40+00:00 [production]
02:27 <LocalisationUpdate> completed (1.23wmf13) at 2014-02-17 02:27:22+00:00 [production]
2014-02-16 §
19:32 <aaron> synchronized php-1.23wmf13/includes/filebackend/SwiftFileBackend.php 'e14a87489d9f65fec85347c8e4a7825576f15be6' [production]
16:03 <ottomata> restarted varnishkafka on esams bits varnishes [production]
15:58 <ottomata> restarted varnishkafka on cp3019 [production]
15:48 <ottomata> starting kafka leader replica election to production load across both brokers evenly. Not yet sure why analytics1022 was the leader for all toppars… [production]
10:48 <apergos> for the record, after the reboot I added back the 10.0.0.45 and ran start-nfs, still not happy [production]
10:11 <ori> labstore4. dmesg: XFS (dm-0): xfs_log_force: error 5 returned. Rebooting. [production]
08:07 <matanya> Labs NFS Issues: cannot open directory .: Stale NFS file handle XFS seems broken again [production]
02:42 <LocalisationUpdate> ResourceLoader cache refresh completed at 2014-02-16 02:42:25+00:00 [production]
02:21 <LocalisationUpdate> completed (1.23wmf14) at 2014-02-16 02:20:57+00:00 [production]
02:11 <LocalisationUpdate> completed (1.23wmf13) at 2014-02-16 02:11:05+00:00 [production]
2014-02-15 §
22:30 <reedy> updated /a/common to {{Gerrit|Id33b8287c}}: Remove 1.23wmf1 through 1.23wmf5 [production]
22:27 <reedy> updated /a/common to {{Gerrit|I04d387adf}}: s1 substitute db1034 for db1055 during schema changes [production]
14:50 <springle> synchronized wmf-config/db-eqiad.php 's1 substitute db1034 for db1055 during schema changes' [production]
05:14 <springle> xtrabackup clone db1055 to db1010 [production]
03:32 <LocalisationUpdate> ResourceLoader cache refresh completed at 2014-02-15 03:31:58+00:00 [production]
02:50 <LocalisationUpdate> completed (1.23wmf14) at 2014-02-15 02:50:08+00:00 [production]
02:29 <LocalisationUpdate> completed (1.23wmf13) at 2014-02-15 02:29:05+00:00 [production]
02:09 <awight_> updated payments from 9e4d8b29581e2465d1acde8d7c2377fa6a8522a6 to ce6233998f4bc0266c2e027c44620a8ba9984681 [production]
01:41 <awight_> updated payments from 9b936320b797bd01c4e61b1cd7c2e15b0820a24b to 9e4d8b29581e2465d1acde8d7c2377fa6a8522a6 [production]
00:56 <ori> updated /a/common to {{Gerrit|I7e014f29e}}: Handle empty lines gracefully [production]
00:52 <awight_> crm rolled back from 9385167c077ccef4c52db2f479b39e5907378b61 to 20684a7e5dd4270e615ca52e6a17b25e880e47c4 [production]
00:50 <awight_> update crm from 20684a7e5dd4270e615ca52e6a17b25e880e47c4 to 9385167c077ccef4c52db2f479b39e5907378b61 [production]
00:45 <ori> updated /a/common to {{Gerrit|Id87a90474}}: Remove unused third/fourth parameters for wikiversions [production]
2014-02-14 §
23:59 <aaron> synchronized php-1.23wmf13/extensions/Math '9e75a1b' [production]
23:32 <Coren> tools Rebooted labstore4 -- XFS done got broken agun [production]
22:53 <kaldari> synchronized php-1.23wmf13/extensions/VectorBeta/ 'sync update for VectorBeta on wmf13' [production]
22:30 <kaldari> synchronized php-1.23wmf14/extensions/VectorBeta/ 'sync update for VectorBeta on wmf14' [production]
22:23 <bd808> manually applied Ie56d3a5 on logstash100[123] hosts and restarted gmond [production]
22:22 <bd808> enabled puppetd on logstash1002 [production]
21:46 <bd808> disabled puppetd on logstash1002 to test ganglia monitor fix [production]
21:43 <bd808> lostash fatalmonitor dashboard working again after restarts to backend [production]
21:30 <bd808> Upgraded and restarted elasticsearch on logstash1003 [production]
21:29 <bd808> Upgraded and restarted elasticsearch on logstash1001 [production]
21:28 <bd808> Upgraded and restarted elasticsearch on logstash1002 [production]
21:25 <awight_> crm updated from bd973432c957b6253f82b7b0590251402e75a0de to 20684a7e5dd4270e615ca52e6a17b25e880e47c4 [production]
20:46 <Reedy> ULSFO down, traffic to Asia etc affected. Being worked on [production]
20:43 <reedy> synchronized wmf-config/InitialiseSettings.php 'Revert Add local interwiki for metawiki' [production]
20:10 <mwalker> updated payments wiki from from bad46aad4b28435d10570aac8173a7f8dca8751d to 9b936320b797bd01c4e61b1cd7c2e15b0820a24b for version stamp and other 1.22.3 changes [production]