|
2013-11-21
§
|
| 15:34 |
<springle> |
bounced pc1001 mysqld. massive spike of writes exhausted innodb txn log slots and wouldn't be killed |
[production] |
| 13:55 |
<hashar> |
restarted Zuul, had some jobs stuck in its queue ({{gerrit|96762}} |
[production] |
| 09:16 |
<paravoid> |
reenable ospfv3 between esams-eqiad |
[production] |
| 08:44 |
<ori-l> |
Synced wmf-config/InitialiseSettings.php "I5e8317db6: Enable recursion-guard log bucket to investigate bug 54193" |
[production] |
| 08:44 |
<ori-l> |
with neon down, scap / syncs not logged to irc |
[production] |
| 08:04 |
<springle> |
rebooted neon, ext3 root fs read only |
[production] |
| 07:52 |
<akosiaris> |
updated all remaining references to sockpuppet and stafford as puppetmaster on wikitech |
[production] |
| 04:58 |
<dr0ptp4kt> |
Wikipedia Zero BDD script run |
[production] |
| 03:20 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Nov 21 03:20:28 UTC 2013 |
[production] |
| 02:32 |
<LocalisationUpdate> |
completed (1.23wmf3) at Thu Nov 21 02:32:33 UTC 2013 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.23wmf4) at Thu Nov 21 02:17:19 UTC 2013 |
[production] |
| 01:39 |
<springle> |
enwiki revision table schema change test run db60 |
[production] |
| 01:24 |
<springle> |
depool db60, db54, db34, db31, db65, db47, db37 for decom and/or shipping. promote db68, db74, db72, db71, db69 to pmtpa shard masters |
[production] |
| 01:18 |
<springle> |
synchronized wmf-config/db-pmtpa.php |
[production] |
| 01:08 |
<awight> |
updated crm from 326854db84becacf808add24a75ca27d1358c5bc to 9979fe9a0f456499ff5be261aa41ca601a88dc13 |
[production] |
| 00:30 |
<AaronSchulz> |
Resynced terbium /usr/local/apache/common-local which was totally empty |
[production] |
| 00:18 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'If961f9365: add dewiki help page for Echo' |
[production] |
| 00:17 |
<ori> |
updated /a/common to {{Gerrit|If961f9365}}: add dewiki help page for Echo |
[production] |
| 00:10 |
<ori-l> |
terbium: symlinked /a/common to /usr/local/apache/common. We should really get rid of the distinction. |
[production] |
|
2013-11-20
§
|
| 23:40 |
<yurik> |
synchronized php-1.23wmf4/extensions/ZeroRatedMobileAccess/ |
[production] |
| 23:37 |
<ori-l> |
puppet.log on terbium: could not set file on ensure: No such file or directory - /usr/local/apache/common/php/extensions/FlaggedRevs/maintenance/wikimedia-periodic-update.sh.puppettmp_8141 at /etc/puppet/manifests/misc/maintenance.pp:155 |
[production] |
| 23:30 |
<yurik> |
synchronized php-1.23wmf3/extensions/ZeroRatedMobileAccess/ |
[production] |
| 23:27 |
<ori-l> |
restarting rsyslog on application servers for I31c76fdde. |
[production] |
| 22:29 |
<bsitu> |
synchronized wmf-config/InitialiseSettings.php 'touch' |
[production] |
| 22:28 |
<bsitu> |
synchronized echowikis.dblist 'Enable Echo and Thanks on dewiki and itwiki' |
[production] |
| 22:23 |
<bsitu> |
synchronized echowikis.dblist 'Enable Echo and Thanks on dewiki and itwiki' |
[production] |
| 22:23 |
<bsitu> |
synchronized wmf-config/InitialiseSettings.php 'Update email footer and help page' |
[production] |
| 22:20 |
<bsitu> |
updated /a/common to {{Gerrit|Id9e3b9a03}}: correct default Echo help page |
[production] |
| 21:47 |
<faidon> |
synchronized wmf-config/CommonSettings.php 'revert CentralNotice CrossWiki Hiding' |
[production] |
| 21:46 |
<faidon> |
updated /a/common to {{Gerrit|I866838bc2}}: Revert "Enable CentralNotice CrossWiki Hiding" |
[production] |
| 21:34 |
<mutante> |
ran checksetup.pl on bugzilla, deploy gerrit 96479, replaces internal errors on WeeklyReport with proper error pages |
[production] |
| 21:12 |
<gwicke> |
deployed Parsoid 20c6afe |
[production] |
| 19:08 |
<yurik> |
synchronized docroot/bits/WikipediaMobileFirefoxOS |
[production] |
| 18:07 |
<akosiaris> |
changed default ferm policy to DROP. antimony/gallium are the first two servers to get it. |
[production] |
| 17:51 |
<^demon|sick> |
gerrit up, forcing replication of all repos |
[production] |
| 17:50 |
<^demon|sick> |
restarting gerrit since replication plugin won't reload |
[production] |
| 17:27 |
<^demon|sick> |
gerrit: disabled replication plugin fix, pending firewall updates. everything's timing out and flooding logs with errors. |
[production] |
| 13:59 |
<hashar> |
jenkins: added slave integration-slave01 with label hasNpm. That is a slave running in labs ( integration-slave01.pmtpa.wmflabs ) |
[production] |
| 12:31 |
<apergos> |
end of an era: shut down ms7 webserver and opening ticket for decom |
[production] |
| 10:14 |
<hashar> |
Jenkins: disabled gearman plugin, Zuul is no more a Gearman server. |
[production] |
| 10:12 |
<hashar> |
restarted again stalled Jenkins |
[production] |
| 10:05 |
<hashar> |
restarted Zuul with non gearman version. |
[production] |
| 10:02 |
<hashar> |
reverting Zuul to last known version: wmf-deploy-20131023 + 6241272...1e3adfd wmf-deploy-20131023 -> master (forced update) |
[production] |
| 09:58 |
<hashar> |
shutting down Zuul and reverting upgrade :-( |
[production] |
| 09:46 |
<hashar> |
restarting Jenkins to update gearman plugin |
[production] |
| 09:38 |
<hashar> |
Zuul upgraded with Gearman support to trigger build in Jenkins. Will monitor for the rest of the morning. |
[production] |
| 09:34 |
<hashar> |
enabling Gearman in Jenkins, making it register with Zuul new version |
[production] |
| 09:34 |
<hashar> |
restarted Zuul with Gearman version |
[production] |
| 09:27 |
<hashar> |
removing all old versions of zuul: rm -fR /usr/local/lib/python2.7/dist-packages/zuul* |
[production] |
| 09:20 |
<hashar> |
gallium temp install of python-pip to be able to ugprade Zuul |
[production] |