|
2013-09-01
§
|
| 23:57 |
<springle> |
running pt-table-sync over db1027 wikis |
[production] |
| 21:29 |
<springle> |
restarted db1027 mysqld. leaving port 3306 blocked while replication recovers |
[production] |
| 20:51 |
<paravoid> |
powercycled db1027, BUG: soft lockup - CPU#14 stuck for 22s! [xfsaild/dm-0:933] |
[production] |
| 20:29 |
<paravoid> |
restarting gerrit |
[production] |
| 17:56 |
<paravoid> |
killed snapshot1004 workers, flooding dberror with wikiadmin auth failures + WikiExporter::dumpFrom slow queries (10' ago) |
[production] |
| 17:52 |
<faidon> |
cleared profiling data |
[production] |
| 17:04 |
<maxsem> |
synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php 'Not the cause, restoring back' |
[production] |
| 16:53 |
<maxsem> |
synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php 'Trying to disable this...' |
[production] |
| 02:07 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sun Sep 1 02:07:18 UTC 2013 |
[production] |
| 02:02 |
<LocalisationUpdate> |
completed (1.22wmf15) at Sun Sep 1 02:02:30 UTC 2013 |
[production] |
| 02:01 |
<LocalisationUpdate> |
completed (1.22wmf14) at Sun Sep 1 02:01:46 UTC 2013 |
[production] |
|
2013-08-30
§
|
| 22:20 |
<mutante> |
after re-pooling search1015 a while ago, now temp. disable search1016 while indices are rebuilt for new wikis |
[production] |
| 21:48 |
<mutante> |
on sodium: Class[Backup::Host] is already defined ... |
[production] |
| 21:45 |
<mutante> |
temp. disabling search1015 in pybal |
[production] |
| 21:27 |
<mutante> |
dumping/importing viwikivoyage and tyvwiki lucene search indices |
[production] |
| 21:23 |
<mutante> |
restarting lucene search indexers on searchidx2 and searchidx1001 |
[production] |
| 20:22 |
<csteipp> |
synchronized php-1.22wmf15/extensions/LiquidThreads 'bug53320' |
[production] |
| 18:00 |
<mutante> |
powercycling sq36 |
[production] |
| 15:25 |
<hashar> |
Jenkins: removing labels hasSlaveScripts hasContintPackages productionSlaves from lanthanum slaves, it caught some jobs |
[production] |
| 14:37 |
<hashar> |
jenkins: lanthanum slave now has 8 executors and labels hasSlaveScripts hasContintPackages productionSlaves |
[production] |
| 10:47 |
<mark> |
reverted |
[production] |
| 10:34 |
<mark> |
Moving esams text squid backend traffic from pmtpa to eqiad |
[production] |
| 08:46 |
<hashar> |
restarting Jenkins for plugins upgrade |
[production] |
| 05:03 |
<olivneh> |
synchronized wmf-config/CommonSettings.php 'Make Flickr API URL protocol-relative for compatibility with HTTPS' |
[production] |
| 02:53 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Aug 30 02:53:46 UTC 2013 |
[production] |
| 02:38 |
<LocalisationUpdate> |
completed (1.22wmf15) at Fri Aug 30 02:38:00 UTC 2013 |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.22wmf14) at Fri Aug 30 02:18:27 UTC 2013 |
[production] |
| 00:18 |
<Ryan_Lane> |
rebooting sq36 |
[production] |
| 00:17 |
<Ryan_Lane> |
switching esams from using pmtpa to using eqiad for carp |
[production] |
|
2013-08-29
§
|
| 23:50 |
<gwicke> |
updated Parsoid to b26a506 |
[production] |
| 23:36 |
<Ryan_Lane> |
removing sq33 from squid configuration, it was decommissioned a while ago |
[production] |
| 21:55 |
<mutante> |
upgrading apparmor packages on iron |
[production] |
| 21:31 |
<mutante> |
powercycled sq36 which was frozen but probably all unrelated to API outage |
[production] |
| 20:57 |
<olivneh> |
synchronized php-1.22wmf15/extensions/NavigationTiming 'Updating NavigationTiming to master (2/2)' |
[production] |
| 20:57 |
<olivneh> |
synchronized php-1.22wmf14/extensions/NavigationTiming 'Updating NavigationTiming to master (1/2)' |
[production] |
| 20:55 |
<mflaschen> |
synchronized php-1.22wmf14/extensions/GuidedTour/ 'Deploy GuidedTour to 1.22wmf14' |
[production] |
| 20:54 |
<olivneh> |
synchronized php-1.22wmf15/includes/RecentChange.php 'I4570473ae' |
[production] |
| 20:51 |
<reedy> |
synchronized php-1.22wmf14/extensions/Wikibase |
[production] |
| 20:50 |
<reedy> |
synchronized php-1.22wmf15/extensions/Wikibase |
[production] |
| 20:31 |
<olivneh> |
synchronized php-1.22wmf15/extensions/NavigationTiming 'Updating NavigationTiming to master (2/2)' |
[production] |
| 20:31 |
<olivneh> |
synchronized php-1.22wmf14/extensions/NavigationTiming 'Updating NavigationTiming to master (1/2)' |
[production] |
| 19:52 |
<Aaron|home> |
Running PurgeDeletedFiles everywhere to bring ceph/swift more in sync with each other and the DB; copy scripts already finished |
[production] |
| 19:52 |
<olivneh> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 19:39 |
<olivneh> |
synchronized wmf-config/CommonSettings.php 'NavigationTiming sample factor to 5000' |
[production] |
| 19:16 |
<reedy> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 19:06 |
<reedy> |
synchronized php-1.22wmf14/extensions/NavigationTiming |
[production] |