|
2014-10-15
§
|
| 14:11 |
<_joe_> |
powercycling mw1205, down since this morning, console blank |
[production] |
| 13:29 |
<Jeff_Green> |
dist-upgrade and reboot indium |
[production] |
| 12:40 |
<hashar> |
disabled/reenabled gearman plugin at https://integration.wikimedia.org/ci/manage |
[production] |
| 12:34 |
<hashar> |
Zuul frozen \\O/ |
[production] |
| 11:38 |
<_joe_> |
depooling mw1114 again |
[production] |
| 10:59 |
<paravoid> |
AMS-IX renumbering: adding second IP, peering with RS1 |
[production] |
| 10:47 |
<_joe_> |
repooled mw1114 with reduced load, using jemalloc with prof_leak enabled for sampling. will depool again soon |
[production] |
| 10:27 |
<_joe_> |
depooling mw1114, stopping puppet for debugging purposes |
[production] |
| 09:24 |
<godog> |
enable container sync for commons containers |
[production] |
| 08:09 |
<hashar> |
restarting Jenkins |
[production] |
| 08:06 |
<hashar> |
Jenkins: upgrading Gearman plugin to Patchset 9 of https://review.openstack.org/#/c/125755/ |
[production] |
| 07:52 |
<springle> |
ongoing schema changes rev_content_(model|format) multiple shards, ok to kill osc_host.sh jobs on terbium in emergency |
[production] |
| 06:58 |
<_joe_> |
restarting hhvm on mw1114 to avoid memory exhaustion |
[production] |
| 05:54 |
<K4-713> |
updated payments to 944596744e0de23fee098 |
[production] |
| 03:34 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Oct 15 03:34:50 UTC 2014 (duration 34m 49s) |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.25wmf3) at 2014-10-15 02:28:00+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-15 02:15:31+00:00 |
[production] |
|
2014-10-14
§
|
| 23:42 |
<catrope> |
Synchronized php-1.25wmf3/extensions/Collection: SWAT (duration: 00m 08s) |
[production] |
| 23:42 |
<catrope> |
Synchronized php-1.25wmf2/extensions/Collection: SWAT (duration: 00m 04s) |
[production] |
| 22:24 |
<hoo> |
Gracefulled apache on mw1075 |
[production] |
| 19:16 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 17s) |
[production] |
| 19:07 |
<reedy> |
Synchronized images/: (no message) (duration: 00m 14s) |
[production] |
| 18:52 |
<Reedy> |
Purged php-1.24wmf18 from mediawiki-appservers |
[production] |
| 18:41 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf3 |
[production] |
| 18:20 |
<reedy> |
Purged l10n cache for 1.25wmf1 |
[production] |
| 18:15 |
<godog> |
stop diamond on virt1000 and zirconium to test |
[production] |
| 17:52 |
<godog> |
conntrack full on virt1000 and zirconium, suspected diamond collector runaway |
[production] |
| 16:33 |
<_joe_> |
repooling mw1114 |
[production] |
| 15:27 |
<_joe_> |
reimaging mw1114 with HHVM - first server in the API pool; depooling and reinstalling now. |
[production] |
| 15:22 |
<marktraceur> |
Synchronized php-1.25wmf3/resources/lib/oojs-ui/: [SWAT] [wmf3] OOjs UI: New pull-through to 837b2f733e to fix a missed dependency (duration: 00m 06s) |
[production] |
| 15:12 |
<marktraceur> |
Synchronized php-1.25wmf3/includes/api/ApiQueryBacklinks.php: [SWAT] [wmf3] API: Fix ApiQueryBacklinks redirlinks (duration: 00m 05s) |
[production] |
| 15:11 |
<marktraceur> |
Synchronized php-1.25wmf2/includes/api/ApiQueryBacklinks.php: [SWAT] [wmf2] API: Fix ApiQueryBacklinks redirlinks (duration: 00m 06s) |
[production] |
| 13:56 |
<godog> |
enable container sync on non-commons sharded containers |
[production] |
| 12:27 |
<hashar> |
Jenkins restarting |
[production] |
| 12:25 |
<hashar> |
Jenkins: upgrading Gearman plugin to fix jobs registrations ( cherry picked https://review.openstack.org/#/c/125755/ and compiled it via maven ). |
[production] |
| 10:38 |
<godog> |
enable container sync on non-sharded originals containers |
[production] |
| 09:46 |
<godog> |
upload python-elasticsearch to trusty-wikimedia |
[production] |
| 09:44 |
<_joe_> |
running sync-common on mw1163 |
[production] |
| 03:34 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Oct 14 03:34:53 UTC 2014 (duration 34m 52s) |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.25wmf3) at 2014-10-14 02:27:23+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-14 02:15:19+00:00 |
[production] |
| 01:42 |
<springle> |
restarted gitblit |
[production] |
|
2014-10-13
§
|
| 22:17 |
<akosiaris> |
ran update-rc.d -f puppetmaster remove on palladium/strontium |
[production] |
| 21:47 |
<akosiaris> |
restarting apache on palladium, strontium |
[production] |
| 20:20 |
<_joe_> |
load test done. HHVM is awesome |
[production] |
| 19:51 |
<_joe_> |
load test on HHVM starting |
[production] |
| 19:13 |
<oblivian> |
Synchronized wmf-config/CommonSettings.php: Serving 5% of anons with HHVM (duration: 00m 12s) |
[production] |
| 17:12 |
<oblivian> |
Synchronized wmf-config/CommonSettings.php: Serving 2% of anons with HHVM (duration: 00m 06s) |
[production] |
| 16:19 |
<_joe_> |
restarting gitlbit, stuck in GC probably |
[production] |
| 14:30 |
<_joe_> |
rolling restart of the ocg cluster |
[production] |