|
2015-11-24
§
|
| 20:01 |
<bblack> |
disabling puppet on neon to avoid monitoring race-condition spam |
[production] |
| 19:30 |
<moritzm> |
restarted hadoop workers on analytics1028 to analytics1055 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk) |
[production] |
| 19:27 |
<bblack> |
misc-cluster: all 4x nodes on new config and pooled |
[production] |
| 19:18 |
<bblack> |
misc-cluster: cp1056+7 pooled (new config), cp1069+70 depooled (old config) |
[production] |
| 18:52 |
<bblack> |
repooled cp1056, depooled cp1057 (misc 2layer) |
[production] |
| 18:38 |
<YuviPanda> |
restarted apache2 on etherpad |
[production] |
| 18:30 |
<bblack> |
depooling cp1056 for misc-cluster 2layer work... |
[production] |
| 17:59 |
<godog> |
nodetool cleanup on restbase1007 |
[production] |
| 17:20 |
<godog> |
stop decommissioning restbase1001, bounce cassandra |
[production] |
| 16:40 |
<thcipriani@tin> |
Synchronized php-1.27.0-wmf.7/resources/lib/oojs-ui/oojs-ui.js: SWAT: OOjs UI: Backport 4fbbc737c86b500c11bbb471ec1001c50ab8853c [[gerrit:255032]] (duration: 00m 28s) |
[production] |
| 16:23 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable es labs replica writes for nlwiki, frwiki and eswiki [[gerrit:255122]] (duration: 00m 27s) |
[production] |
| 16:16 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Fix article-recommender-1 campaign [[gerrit:255069]] (duration: 00m 27s) |
[production] |
| 16:10 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Third QuickSurveys external survey [[gerrit:254908]] (duration: 00m 30s) |
[production] |
| 16:08 |
<paravoid> |
starting pybal on lvs3001 again, testing is over |
[production] |
| 16:06 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all new accounts on eswiki [[gerrit:250473]] (duration: 00m 46s) |
[production] |
| 15:38 |
<moritzm> |
restarted analytics1030 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk) |
[production] |
| 15:33 |
<hashar> |
enabled puppet on both gallium and scandium |
[production] |
| 15:27 |
<hashar> |
disabling puppet on both gallium and scandium |
[production] |
| 14:24 |
<godog> |
upgrade diamond to 3.5-4 on precise hosts in eqiad |
[production] |
| 14:22 |
<godog> |
upgrade diamond to 3.5-4 on precise hosts in ulsfo |
[production] |
| 13:56 |
<godog> |
upgrade diamond to 3.5-4 on precise hosts in esams |
[production] |
| 12:18 |
<paravoid> |
stopping pybal on lvs3001, switching its traffic to lvs3003 |
[production] |
| 11:48 |
<paravoid> |
rebooting lvs3003 with linux 4.2.6-1~bpo8+wmf1 |
[production] |
| 10:28 |
<jynus> |
dropping user_daily_contribs view from labsdb hosts |
[production] |
| 10:07 |
<godog> |
stop ircecho on neon temporarily while https://gerrit.wikimedia.org/r/#/c/255094/ applies |
[production] |
| 08:24 |
<jynus> |
dbstore1002 (analytics-slave) mysql is not responding, forcing restart |
[production] |
| 05:58 |
<_joe_> |
powercycling ms-be1010, xfs kernel lockups |
[production] |
| 02:26 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 34s) |
[production] |
| 00:36 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.7/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/255053/ (duration: 00m 28s) |
[production] |
| 00:26 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable commonswiki for elasticsearch labs replica writes (duration: 00m 28s) |
[production] |
| 00:15 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable wikidatawiki for elasticsearch labs replica (duration: 00m 27s) |
[production] |
|
2015-11-23
§
|
| 23:55 |
<maxsem@tin> |
Synchronized portals: (no message) (duration: 00m 28s) |
[production] |
| 23:08 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable elasticsearch labs replica for enwiki and dewiki (duration: 00m 28s) |
[production] |
| 21:16 |
<hoo> |
Updated Wikidata's property suggester with data from today's json dump |
[production] |
| 21:15 |
<jynus> |
enforcing limits on running queries on all labsdb databases |
[production] |
| 20:47 |
<jynus> |
restarting labsdb1002, hopfully for the last time |
[production] |
| 17:54 |
<jynus> |
temporarily stopping labsdb1002 to solve replication issues |
[production] |
| 17:32 |
<Coren> |
emergency hotfix on labstore1001 (apt-get remove nfsd-ldap) as it appears to not work in all situations. |
[production] |
| 17:30 |
<ori> |
restarting hhvm on mw1122 after lock-up; backtrace saved as /tmp/hhvm.32016.bt. |
[production] |
| 17:28 |
<thcipriani@tin> |
Finished scap: SWAT: Update CentralNotice [[gerrit:254865]] (duration: 26m 31s) |
[production] |
| 17:02 |
<thcipriani@tin> |
Started scap: SWAT: Update CentralNotice [[gerrit:254865]] |
[production] |
| 16:44 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: noindex user namespace on en.wikipedia.org [[gerrit:237330]] (duration: 00m 28s) |
[production] |
| 16:43 |
<jynus> |
restarting mysqld on labsdb1003 again |
[production] |
| 16:38 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: End first mobile QuickSurveys campaign [[gerrit:254831]] (duration: 00m 28s) |
[production] |
| 16:32 |
<thcipriani@tin> |
Synchronized php-1.27.0-wmf.7/includes/specials/SpecialWatchlist.php: SWAT: Special:Watchlist: Add user preference to "Show last" options, fix float comparison [[gerrit:254880]] (duration: 00m 29s) |
[production] |
| 16:10 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable article-recommender-1 campaign as default [[gerrit:254117]] (duration: 00m 52s) |
[production] |
| 15:56 |
<paravoid> |
rebooted lvs3003 back with (an updated) 3.19 |
[production] |
| 15:53 |
<_joe_> |
live-resizing /var/lib/wdqs on wdqs1002 |
[production] |
| 15:15 |
<Coren> |
NFS server on labstore1001 restarted with no issues. |
[production] |
| 15:11 |
<Coren> |
Restarting NFS daemon on labstore1001 w/ the new LDAP shim. |
[production] |