|
2014-04-16
§
|
| 11:39 |
<hashar> |
Upgraded Zuul to wmf-deploy-20140416-3 (bring in a84f0e4 - "Make queue processing more efficient" which was much needed) |
[production] |
| 11:29 |
<hashar> |
upgraded Zuul to wmf-deploy-20140416-2 |
[production] |
| 11:16 |
<mutante> |
virt5-11 removing from icinga |
[production] |
| 11:03 |
<mutante> |
virt5-11 revoked puppet certs and salt keys |
[production] |
| 10:56 |
<mutante> |
stopping puppet on virt5-11 |
[production] |
| 10:47 |
<hashar> |
Upgraded Zuul on gallium to wmf-deploy-20140416 (depends on python-voluptuous 0.7+ , Alexandros packaged 0.8.2 which I manually installed to validate). |
[production] |
| 09:26 |
<mutante> |
disabling mw1163 in pybal |
[production] |
| 07:03 |
<mutante> |
zirconium - upgrading apache2, php5 packages |
[production] |
| 06:07 |
<springle> |
stop mysqld on db38 (x1) for decom |
[production] |
| 03:46 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Apr 16 03:46:23 UTC 2014 (duration 46m 22s) |
[production] |
| 02:55 |
<LocalisationUpdate> |
completed (1.23wmf22) at 2014-04-16 02:55:28+00:00 |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.23wmf21) at 2014-04-16 02:28:01+00:00 |
[production] |
| 00:43 |
<K4-713> |
updated listener credentials on thulium |
[production] |
|
2014-04-15
§
|
| 23:25 |
<mwalker> |
synchronized wmf-config/abusefilter.php '{{gerrit|126168}} more abuse filter configuration fun' |
[production] |
| 23:21 |
<mwalker> |
Finished scap: Configuration change {{gerrit|126163}} and MultimediaViewer {{gerrit|126158}} (duration: 02m 15s) |
[production] |
| 23:19 |
<mwalker> |
Started scap: Configuration change {{gerrit|126163}} and MultimediaViewer {{gerrit|126158}} |
[production] |
| 23:08 |
<mwalker> |
Finished scap: Configuration changes, {{gerrit|113656}}, {{gerrit|121834}}, {{gerrit|126065}} (duration: 03m 11s) |
[production] |
| 23:05 |
<mwalker> |
Started scap: Configuration changes, {{gerrit|113656}}, {{gerrit|121834}}, {{gerrit|126065}} |
[production] |
| 23:01 |
<hashar> |
restarting Zuul to clear leaked file descriptor (know issue, fixed upstream) |
[production] |
| 22:12 |
<awight> |
crm updated from e3f285984f786ca9a05ded1662aba415b0259856 to 7dafce5f8fe265fb0ab6c96e01e59fc4362ea5b4 |
[production] |
| 21:51 |
<manybubbles> |
restarting elastic1009 again |
[production] |
| 21:39 |
<hashar> |
jenkins /var/lib/git cleaned up on gallium |
[production] |
| 21:16 |
<manybubbles> |
restarting elastic1009 to test performance changes. cluster will go yellow for a few minutes. might go red (wikitech is busted) |
[production] |
| 21:15 |
<hashar> |
Jenkins is processing jobs again |
[production] |
| 21:14 |
<hashar> |
cleared /tmp/ on integration-slave1002 (filled up by hhvm job, known issue, bug filled already) |
[production] |
| 21:12 |
<hashar> |
Zuul locked again :/ Unpooling and repooling Jenkins slaves. |
[production] |
| 19:50 |
<RoanKattouw> |
Restarting stuck Jenkins |
[production] |
| 19:31 |
<manybubbles> |
setting refresh interval on elasticsearch indexes to 30s to test effect on load |
[production] |
| 19:24 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 19:20 |
<reedy> |
synchronized php-1.23wmf22/includes/PrefixSearch.php 'I82b5ca65864099c180d915055c43e6839bd4f4a2' |
[production] |
| 19:07 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikisources back to 1.23wmf22 |
[production] |
| 19:07 |
<ottomata> |
reinstalling elastic1010 |
[production] |
| 19:07 |
<reedy> |
synchronized php-1.23wmf22/extensions/ProofreadPage |
[production] |
| 18:41 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikisources back to 1.23wmf21 due to ProofreadPage fatal |
[production] |
| 18:36 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.23wmf22 |
[production] |
| 17:09 |
<paravoid> |
stopped pybal on lvs1005 |
[production] |
| 17:06 |
<cmjohnson1> |
fixing lvs1005 eth1 cable |
[production] |
| 16:56 |
<cmjohnson1> |
mw1057 replacing ethernet cable |
[production] |
| 16:50 |
<manybubbles> |
raised "new generation" size on elastic1009 to test a performance theory |
[production] |
| 16:50 |
<cmjohnson1> |
mw1093 replacing ethernet cable |
[production] |
| 16:40 |
<cmjohnson1> |
replacing eth cable on mw1193 |
[production] |
| 16:31 |
<hashar> |
... all Jenkins jobs are using /srv/ssd/gerrit instead |
[production] |
| 16:30 |
<hashar> |
gallium had two Gerrit replications streams, one of them got removed {{gerrit|122419}} thus deleting the target directories under /var/lib/git |
[production] |
| 16:22 |
<cmjohnson1> |
shutting down mw1163 to replace DIMM |
[production] |
| 16:18 |
<cmjohnson1> |
swapping bad disk slot 4 on dataset1001 |
[production] |
| 16:13 |
<paravoid> |
moving ms-fe3xxx/ms-be3xxx to private1-esams |
[production] |
| 16:06 |
<ottomata> |
reinstalling elastic1009 |
[production] |
| 15:21 |
<anomie> |
synchronized php-1.23wmf21/extensions/Flow 'SWAT: Flow: Prevent logspam on enwiki 125930' |
[production] |
| 15:13 |
<anomie> |
synchronized php-1.23wmf21/extensions/Flow 'SWAT: Flow: Prevent logspam on enwiki 125930' |
[production] |
| 15:02 |
<mutante> |
DNS update - removing Tampa service IPs |
[production] |