|
2014-10-16
§
|
| 14:08 |
<_joe_> |
depooling mw1189 from the api pool, reimaging with hhvm |
[production] |
| 10:11 |
<hashar> |
Updated our Jenkins Job Builder forked repository ( ee80dbc..7ad4386 ). No job configuration impact. |
[production] |
| 09:52 |
<paravoid> |
rebooting ms-be1003, sdn3/xfs troubles |
[production] |
| 09:30 |
<paravoid> |
powercycling tmh1002, unresponsive, stuck, no vga output |
[production] |
| 08:44 |
<godog> |
powercycle ms-be1007, no ssh and no console |
[production] |
| 07:53 |
<hashar> |
Jenkins: upgrading PHPUnit from 3.7.28 to 3.7.37 {{gerrit|164683}} [https://lists.wikimedia.org/pipermail/wikitech-l/2014-October/079049.html wikitech-l announce] |
[production] |
| 04:44 |
<springle> |
xtrabackup clone es1004 to es2001 |
[production] |
| 04:24 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Oct 16 04:24:21 UTC 2014 (duration 24m 20s) |
[production] |
| 02:46 |
<LocalisationUpdate> |
completed (1.25wmf3) at 2014-10-16 02:46:46+00:00 |
[production] |
| 02:25 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-16 02:24:55+00:00 |
[production] |
| 01:31 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1061 (duration: 00m 07s) |
[production] |
| 00:49 |
<Krinkle> |
Zuul queue made unstuck by fixing the clogged build (see bug 72113) |
[production] |
| 00:14 |
<Krinkle> |
Jenkins queue is stuck (99% free executors, but it's not running any of Zuul's pending jobs) |
[production] |
|
2014-10-15
§
|
| 23:10 |
<maxsem> |
Synchronized php-1.25wmf2/extensions/MobileFrontend/: (no message) (duration: 00m 04s) |
[production] |
| 23:04 |
<maxsem> |
Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/166886/ (duration: 00m 05s) |
[production] |
| 21:37 |
<ori> |
restarting hhvm on mw1114, this time with luasandbox |
[production] |
| 21:20 |
<hoo> |
Gracefulled apache on mw1115 |
[production] |
| 21:06 |
<ejegg> |
updated crm from 05e5388df34059c651223d53fb2986ac1c39a2d9 to d6a75b6df4482de61da372fa653902db7ca12766 |
[production] |
| 21:02 |
<legoktm_> |
running mwscript extensions/CentralAuth/maintenance/migrateAccount.php on terbium for broken accounts (bug 61876) |
[production] |
| 20:49 |
<mutante> |
deleting/shredding ishmael cert/keys from neon |
[production] |
| 20:41 |
<hoo> |
Deleted 147 orphan wb_terms entries (bug 71914) |
[production] |
| 20:40 |
<ori> |
restarting apache on mw1115 to test luasandbox |
[production] |
| 20:25 |
<yurik> |
Synchronized wmf-config/mobile.php: Disable font for ZeroBanner (duration: 01m 05s) |
[production] |
| 20:17 |
<hoo> |
Deleted ten orphan wb_entity_per_page rows on wikidata |
[production] |
| 20:17 |
<yurik> |
Synchronized php-1.25wmf3/extensions/ZeroPortal: updatidng ZeroPortal to master (duration: 01m 11s) |
[production] |
| 20:13 |
<arlolra> |
restarted ocg service |
[production] |
| 20:02 |
<yurik> |
Synchronized php-1.25wmf2/extensions/ZeroPortal: updatidng ZeroPortal to master (duration: 01m 15s) |
[production] |
| 19:47 |
<yurik> |
Synchronized php-1.25wmf3/extensions/ZeroBanner/: Latest ZeroBanner (duration: 01m 11s) |
[production] |
| 19:37 |
<yurik> |
Synchronized php-1.25wmf2/extensions/ZeroBanner/: Latest ZeroBanner (duration: 01m 07s) |
[production] |
| 19:05 |
<ottomata> |
deployed webstatscollector 0.4 on oxygen (filter) and gadolinium (collector) |
[production] |
| 18:46 |
<ori> |
adjusting pybal weight for mw1114 back up to 20 to confirm that leak is in luasandbox |
[production] |
| 17:57 |
<ori_> |
installed lua5.1 on mw1114 so i can switch scribunto to luastandalone and thus potentially isolate the leak to luasandbox |
[production] |
| 15:38 |
<andrewbogott> |
running sync-common on virt1000 |
[production] |
| 15:36 |
<marktraceur> |
Synchronized php-1.25wmf3/extensions/OpenStackManager/: [SWAT] [wmf3] Make list=novainstances available to anons (duration: 00m 06s) |
[production] |
| 15:36 |
<marktraceur> |
Synchronized php-1.25wmf2/extensions/OpenStackManager/: [SWAT] [wmf2] Make list=novainstances available to anons (duration: 00m 05s) |
[production] |
| 15:22 |
<paravoid> |
AMS-IX renumbering: remove old IP from interface, migration over; > 75% of total peers migrated, accounting for much more bandwidth/routes |
[production] |
| 15:13 |
<marktraceur> |
Synchronized php-1.25wmf3/extensions/VisualEditor/modules/ve-mw/ui: [SWAT] [wmf3] Update OOjs UI to v0.1.0-pre (d74a46ca6a) and VisualEditor-MediaWiki to Ie06056b (duration: 00m 05s) |
[production] |
| 15:12 |
<marktraceur> |
Synchronized php-1.25wmf3/resources/lib/oojs-ui: [SWAT] [wmf3] Update OOjs UI to v0.1.0-pre (d74a46ca6a) and VisualEditor-MediaWiki to Ie06056b (duration: 00m 06s) |
[production] |
| 14:32 |
<paravoid> |
AMS-IX renumbering: move all remaining ASNs to the new space |
[production] |
| 14:20 |
<Coren> |
Not reimaging mw1035 after all; hhvm is in our base, killing our ramz. |
[production] |
| 14:16 |
<paravoid> |
AMS-IX renumbering: peering with (renumbered) top-10 ASNs + ASNs with large number of prefixes |
[production] |
| 14:12 |
<Coren> |
reimaging mw1035 for great justice!!! (HHVM) |
[production] |
| 14:11 |
<_joe_> |
powercycling mw1205, down since this morning, console blank |
[production] |
| 13:29 |
<Jeff_Green> |
dist-upgrade and reboot indium |
[production] |
| 12:40 |
<hashar> |
disabled/reenabled gearman plugin at https://integration.wikimedia.org/ci/manage |
[production] |
| 12:34 |
<hashar> |
Zuul frozen \\O/ |
[production] |
| 11:38 |
<_joe_> |
depooling mw1114 again |
[production] |
| 10:59 |
<paravoid> |
AMS-IX renumbering: adding second IP, peering with RS1 |
[production] |
| 10:47 |
<_joe_> |
repooled mw1114 with reduced load, using jemalloc with prof_leak enabled for sampling. will depool again soon |
[production] |
| 10:27 |
<_joe_> |
depooling mw1114, stopping puppet for debugging purposes |
[production] |