|
2015-11-25
§
|
| 08:08 |
<ori@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: I9bea66df: Update job queue config to use 2 rdb1007 instances (duration: 00m 28s) |
[production] |
| 08:06 |
<akosiaris> |
update puppet compilers fact database |
[production] |
| 03:44 |
<TimStarling> |
on palladium ran puppet post-merge hook manually after puppet-merge failed with "error: Ref refs/remotes/origin/production is at 380706c5bd2f47d140ffa183dd22fe920927af7e but expected 897486120d645f1c30dcf68bfb3198e2f39ec639" |
[production] |
| 03:35 |
<krinkle@tin> |
Synchronized portals/prod/wikipedia.org/index.html: Fix Uncaught TypeError: innerHTML() is not a function (duration: 00m 28s) |
[production] |
| 03:12 |
<krinkle@tin> |
Synchronized wmf-config/CommonSettings.php: I4e21eda0f3 (duration: 01m 06s) |
[production] |
| 02:40 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 06m 50s) |
[production] |
| 02:00 |
<l10nupdate@tin> |
LocalisationUpdate failed: git pull of core failed |
[production] |
| 01:31 |
<krinkle@tin> |
Synchronized php-1.27.0-wmf.7/StartProfiler.php: create symlink (duration: 01m 10s) |
[production] |
| 01:03 |
<mutante> |
re-armed keyholders on mira |
[production] |
| 00:46 |
<mutante> |
starting redis on rdb1007 (it died and was status: dead) |
[production] |
| 00:40 |
<mutante> |
tin: fixing l10nupdate UID (997->10002), file ownership |
[production] |
| 00:37 |
<ori> |
Migrating rdb1007 from redis::legacy to redis::instance; will involve a service restart. |
[production] |
| 00:20 |
<mutante> |
rebooting mira |
[production] |
| 00:14 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/255276/ (duration: 00m 28s) |
[production] |
| 00:09 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/254443/ (duration: 00m 28s) |
[production] |
|
2015-11-24
§
|
| 23:43 |
<mutante> |
fixinx l10nupdate UID / file ownership on mira |
[production] |
| 23:40 |
<csteipp> |
deployed patch for T119309 |
[production] |
| 23:36 |
<ostriches> |
phabricator: restarted daemons, were complaining about out of sync config |
[production] |
| 21:39 |
<krinkle@tin> |
Synchronized wmf-config/CommonSettings.php: I80d38773057 (duration: 00m 28s) |
[production] |
| 20:31 |
<mutante> |
gnt-instance reboot seaborgium.wikimedia.org |
[production] |
| 20:01 |
<bblack> |
disabling puppet on neon to avoid monitoring race-condition spam |
[production] |
| 19:30 |
<moritzm> |
restarted hadoop workers on analytics1028 to analytics1055 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk) |
[production] |
| 19:27 |
<bblack> |
misc-cluster: all 4x nodes on new config and pooled |
[production] |
| 19:18 |
<bblack> |
misc-cluster: cp1056+7 pooled (new config), cp1069+70 depooled (old config) |
[production] |
| 18:52 |
<bblack> |
repooled cp1056, depooled cp1057 (misc 2layer) |
[production] |
| 18:38 |
<YuviPanda> |
restarted apache2 on etherpad |
[production] |
| 18:30 |
<bblack> |
depooling cp1056 for misc-cluster 2layer work... |
[production] |
| 17:59 |
<godog> |
nodetool cleanup on restbase1007 |
[production] |
| 17:20 |
<godog> |
stop decommissioning restbase1001, bounce cassandra |
[production] |
| 16:40 |
<thcipriani@tin> |
Synchronized php-1.27.0-wmf.7/resources/lib/oojs-ui/oojs-ui.js: SWAT: OOjs UI: Backport 4fbbc737c86b500c11bbb471ec1001c50ab8853c [[gerrit:255032]] (duration: 00m 28s) |
[production] |
| 16:23 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable es labs replica writes for nlwiki, frwiki and eswiki [[gerrit:255122]] (duration: 00m 27s) |
[production] |
| 16:16 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Fix article-recommender-1 campaign [[gerrit:255069]] (duration: 00m 27s) |
[production] |
| 16:10 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Third QuickSurveys external survey [[gerrit:254908]] (duration: 00m 30s) |
[production] |
| 16:08 |
<paravoid> |
starting pybal on lvs3001 again, testing is over |
[production] |
| 16:06 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all new accounts on eswiki [[gerrit:250473]] (duration: 00m 46s) |
[production] |
| 15:38 |
<moritzm> |
restarted analytics1030 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk) |
[production] |
| 15:33 |
<hashar> |
enabled puppet on both gallium and scandium |
[production] |
| 15:27 |
<hashar> |
disabling puppet on both gallium and scandium |
[production] |
| 14:24 |
<godog> |
upgrade diamond to 3.5-4 on precise hosts in eqiad |
[production] |
| 14:22 |
<godog> |
upgrade diamond to 3.5-4 on precise hosts in ulsfo |
[production] |
| 13:56 |
<godog> |
upgrade diamond to 3.5-4 on precise hosts in esams |
[production] |
| 12:18 |
<paravoid> |
stopping pybal on lvs3001, switching its traffic to lvs3003 |
[production] |
| 11:48 |
<paravoid> |
rebooting lvs3003 with linux 4.2.6-1~bpo8+wmf1 |
[production] |
| 10:28 |
<jynus> |
dropping user_daily_contribs view from labsdb hosts |
[production] |
| 10:07 |
<godog> |
stop ircecho on neon temporarily while https://gerrit.wikimedia.org/r/#/c/255094/ applies |
[production] |
| 08:24 |
<jynus> |
dbstore1002 (analytics-slave) mysql is not responding, forcing restart |
[production] |
| 05:58 |
<_joe_> |
powercycling ms-be1010, xfs kernel lockups |
[production] |
| 02:26 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 34s) |
[production] |
| 00:36 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.7/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/255053/ (duration: 00m 28s) |
[production] |
| 00:26 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable commonswiki for elasticsearch labs replica writes (duration: 00m 28s) |
[production] |