|
2016-01-29
§
|
| 15:32 |
<akosiaris> |
remove all networking configuration from asw-b-eqiad switch for nas1001-a, nas1001-b. Leave just descriptions |
[production] |
| 15:21 |
<bblack> |
upgrading packages (incl kernel) on esams cache hosts (cp3xxx) (codfw, ulsfo already done) |
[production] |
| 15:11 |
<akosiaris> |
powering off nas1001-a.eqiad.wmnet. https://phabricator.wikimedia.org/T124156 |
[production] |
| 15:08 |
<akosiaris> |
powering off nas1001-b.eqiad.wmnet. https://phabricator.wikimedia.org/T124156 |
[production] |
| 15:01 |
<elukey> |
re-enabled puppet on analytics1027 |
[production] |
| 14:39 |
<elukey> |
stopped kafka (service) on kafka1012 (the host that caused the outage) |
[production] |
| 14:24 |
<moritzm> |
rebooting bohrium for kernel update |
[production] |
| 14:03 |
<_joe_> |
installing the new hhvm package on all the codfw appserver |
[production] |
| 13:43 |
<_joe_> |
installing the new HHVM package to the canary appservers (main and api) |
[production] |
| 12:30 |
<paravoid> |
force-rebooting pollux |
[production] |
| 11:43 |
<_joe_> |
uploaded hhvm_3.6.5+dfsg1-1+wm8 to trusty-wikimedia |
[production] |
| 11:22 |
<moritzm> |
rolling restart of swift in codfw |
[production] |
| 11:14 |
<elukey> |
disabled puppet on analytics1027 due to issues with Camus and HDFS |
[production] |
| 10:17 |
<moritzm> |
rolling restart of swift in esams |
[production] |
| 02:32 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Jan 29 02:32:56 UTC 2016 (duration 7m 28s) |
[production] |
| 02:25 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.11) (duration: 10m 40s) |
[production] |
| 01:31 |
<ori@mira> |
Synchronized wmf-config: I83da57cf: Enable persistent redis connections for job runners (duration: 01m 11s) |
[production] |
| 01:03 |
<krenair@mira> |
Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/267186/ (duration: 01m 09s) |
[production] |
| 01:01 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/265292/ (duration: 01m 14s) |
[production] |
| 00:57 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267071/ (duration: 01m 11s) |
[production] |
| 00:53 |
<krenair@mira> |
Synchronized wmf-config/CirrusSearch-production.php: https://gerrit.wikimedia.org/r/#/c/266995/ (duration: 01m 11s) |
[production] |
| 00:50 |
<yurik> |
synced latest graphoid |
[production] |
| 00:49 |
<krenair@mira> |
Synchronized php-1.27.0-wmf.11/extensions/MobileFrontend/resources/skins.minerva.editor/init.js: https://gerrit.wikimedia.org/r/#/c/267168/ (duration: 01m 12s) |
[production] |
| 00:45 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267053/ (duration: 01m 10s) |
[production] |
| 00:43 |
<krenair@mira> |
Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/#/c/267053/ (duration: 01m 10s) |
[production] |
| 00:42 |
<krenair@mira> |
Synchronized tests/cirrusTest.php: https://gerrit.wikimedia.org/r/#/c/267053/ (duration: 01m 11s) |
[production] |
| 00:35 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267025/ (duration: 01m 12s) |
[production] |
| 00:25 |
<krenair@mira> |
Synchronized php-1.27.0-wmf.11/extensions/Graph/modules/graph2.js: https://gerrit.wikimedia.org/r/#/c/267065/ (duration: 01m 11s) |
[production] |
| 00:17 |
<krenair@mira> |
Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/267060/ (duration: 01m 12s) |
[production] |
| 00:02 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267189/2 (duration: 01m 11s) |
[production] |
|
2016-01-28
§
|
| 23:51 |
<mutante> |
caesium - stop puppet, shutdown server, remove from icinga, clean puppet cert ... |
[production] |
| 23:46 |
<TimStarling> |
on ruthenium installing build dependencies and compiling uprightdiff for test |
[production] |
| 23:20 |
<ori@mira> |
Synchronized php-1.27.0-wmf.11/includes/api/ApiStashEdit.php: Ia4196eba9: Add ParserOutputStashForEdit hook for extension cache warming (duration: 01m 10s) |
[production] |
| 23:17 |
<tgr@mira> |
Synchronized php-1.27.0-wmf.11/includes/session/SessionManager.php: T125161 (duration: 01m 11s) |
[production] |
| 22:58 |
<ottomata> |
restoring MobileWebSectionUsage_14321266 from db1047 to dbstore1002 using mysqlimport |
[production] |
| 22:23 |
<bblack> |
starting cache_mobile->cache_text conversion in eqiad - https://phabricator.wikimedia.org/T109286 |
[production] |
| 22:09 |
<bblack> |
eqiad pybal->etcd conversion done |
[production] |
| 22:01 |
<dduvall@mira> |
Synchronized php-1.27.0-wmf.11/extensions/WikimediaEvents/WikimediaEventsHooks.php: deploying fix for T125151 (duration: 01m 15s) |
[production] |
| 21:59 |
<mutante> |
releases.wm.org - switched backend to bromine |
[production] |
| 21:58 |
<bblack> |
converting active eqiad LVS/pybal to etcd |
[production] |
| 21:55 |
<mutante> |
caesium - stopped apache |
[production] |
| 21:31 |
<ori@mira> |
Synchronized php-1.27.0-wmf.11/extensions/AbuseFilter: I13fcc3ce4: Updated mediawiki/core Project: mediawiki/extensions/AbuseFilter 19baa3b6e51b8fe6baf6e3ce7e590060e8e6eec9 (duration: 01m 11s) |
[production] |
| 21:27 |
<bblack> |
converting backup/inactive eqiad LVS/pybal to etcd |
[production] |
| 21:16 |
<dduvall@mira> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.11 |
[production] |
| 20:54 |
<mutante> |
sca1001 - stop mathoid,graphoid,citoid |
[production] |
| 20:52 |
<mutante> |
sca1002 - stop mathoid,graphoid,citoid |
[production] |
| 20:50 |
<dduvall@mira> |
Synchronized php-1.27.0-wmf.11: syncing 1.27.0-wmf.11 for T125114 and https://gerrit.wikimedia.org/r/#/c/267128/ (duration: 03m 30s) |
[production] |
| 20:25 |
<bblack> |
depool -> reboot cp4008 (ulsfo text, trying new kernel with live traffic) |
[production] |
| 20:00 |
<bblack> |
depool -> reboot cp4011 (ulsfo mobile, currently unused for traffic - testing local conftool-scripts depool + new kernel) |
[production] |
| 19:55 |
<ori@mira> |
Synchronized wmf-config: Iea2573ccfbe: Revert "Autopromotion: remove deprecated onView event, fix INGROUPS" (duration: 02m 13s) |
[production] |