|
2015-10-30
§
|
| 22:01 |
<mutante> |
labvirt1010: salt-minion always in "stop/waiting", doesn't restart |
[production] |
| 21:54 |
<mutante> |
labvirt1010 - start salt |
[production] |
| 21:41 |
<ori> |
Removing MediaWiki.xhprof.* from graphite{1,2}001 |
[production] |
| 21:41 |
<ori@tin> |
Synchronized wmf-config/StartProfiler.php: I0bfa21b5: Disable xhprof profiling to relieve storage pressure on graphite (duration: 00m 18s) |
[production] |
| 18:56 |
<jynus> |
disabling puppet and restarting mysql on es2008, es2009 and es2010- downtime planned for 2 hours - no production impact |
[production] |
| 17:33 |
<robh> |
bismuth has no console output, appears locked up in OS, rebooting via mgmt |
[production] |
| 17:22 |
<ori@tin> |
Synchronized docroot and w: (no message) (duration: 00m 18s) |
[production] |
| 16:56 |
<cmjohnson> |
reinstalling mw1083 https://phabricator.wikimedia.org/T116184 |
[production] |
| 15:35 |
<moritzm> |
updated db205[6-8] to the 3.19 kernel |
[production] |
| 14:38 |
<chasemp> |
nobelium in downtime -- this is a temporary test host for discovery and is not ops actionable |
[production] |
| 13:47 |
<Krenair> |
revert commit for https://gerrit.wikimedia.org/r/#/c/249971/ on tin |
[production] |
| 13:47 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.4/extensions/VisualEditor/VisualEditor.hooks.php: https://gerrit.wikimedia.org/r/#/c/249999/ (duration: 00m 17s) |
[production] |
| 11:57 |
<paravoid> |
upgrading tor to latest stable on radium |
[production] |
| 11:31 |
<paravoid> |
ignoring asw-d-eqiad on librenms |
[production] |
| 11:25 |
<_joe_> |
powercycling dataset1001, stuck in some kernel task, unable to login from console |
[production] |
| 10:42 |
<dcausse> |
elastic in eqiad setting indexing trace threshold to -1 for commons, ruwiki, frwiki and itwiki |
[production] |
| 10:30 |
<dcausse> |
elastic in eqiad wide settings for indexing slow threshold are ineffective, trying to set index settings (T117181) |
[production] |
| 09:18 |
<dcausse> |
elastic in eqiad setting index.indexing.slowlog.threshold.index.trace to -1 |
[production] |
| 08:46 |
<_joe_> |
removed production-search-eqiad_index_indexing_slowlog.log.{7,8} on es1008 |
[production] |
| 06:24 |
<krinkle@tin> |
Synchronized php-1.27.0-wmf.4/includes/jobqueue/JobQueueRedis.php: (no message) (duration: 00m 18s) |
[production] |
| 05:11 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Oct 30 05:11:08 UTC 2015 (duration 11m 7s) |
[production] |
| 04:19 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: I1d0337b58: ScribuntoSlowFunctionThreshold: 0.90 -> 0.99 (duration: 00m 17s) |
[production] |
| 04:05 |
<ori@tin> |
Synchronized php-1.27.0-wmf.4/extensions/Scribunto/common/Hooks.php: I60b9eb617: When logging perf stats, include wfWikiId() in metric key (duration: 00m 18s) |
[production] |
| 02:30 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-30 02:30:50+00:00 |
[production] |
| 02:27 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 13s) |
[production] |
| 00:48 |
<tgr@tin> |
Synchronized php-1.27.0-wmf.4/includes/specials/SpecialUserlogin.php: T117027 (duration: 00m 18s) |
[production] |
| 00:34 |
<mutante> |
labvirt1010 - started salt-minion (no such process) |
[production] |
| 00:34 |
<mutante> |
mw1193 - restarted HHVM (socket timeout) |
[production] |
| 00:34 |
<mutante> |
wdqs1001 - restarted NTP (unknown offset) |
[production] |
| 00:09 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.4/extensions/Flow/: Fix regressions from SWAT (duration: 00m 20s) |
[production] |
| 00:08 |
<mutante> |
radium - back in service |
[production] |
| 00:01 |
<bd808> |
Restarted logstash on logstash1003 again. The first try apparently didn't take |
[production] |
|
2015-10-29
§
|
| 23:44 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: Ibac0d60bd: Disable ScribuntoGatherFunctionStats (duration: 00m 17s) |
[production] |
| 23:32 |
<ebernhardson@tin> |
Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/249922 (duration: 00m 17s) |
[production] |
| 23:32 |
<bd808> |
Restarted logstash on logstash1003; died with OOM error |
[production] |
| 23:24 |
<ori@tin> |
Synchronized php-1.27.0-wmf.4/extensions/Scribunto/engines/LuaSandbox/Engine.php: I69e9218 (duration: 00m 18s) |
[production] |
| 23:21 |
<mutante> |
radium: scheduled downtime, reinstalling |
[production] |
| 23:19 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/: Fix unwritable cluster errors (duration: 00m 19s) |
[production] |
| 23:18 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.4/extensions/Flow: Fix CAPTCHA rendering in RTL languages (duration: 00m 19s) |
[production] |
| 23:17 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.4/extensions/Scribunto/: Make the percentile threshold for slow function stats configurable (duration: 00m 18s) |
[production] |
| 23:16 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.4/resources/src/mediawiki.special/mediawiki.special.search.css: SWAT: styling tweaks for inline interwiki search (duration: 00m 18s) |
[production] |
| 22:33 |
<tgr@tin> |
Synchronized php-1.27.0-wmf.4/extensions/ZeroBanner/includes/ZeroSpecialPage.php: T116821 (duration: 00m 17s) |
[production] |
| 22:31 |
<tgr@tin> |
Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066, fixes some exceptions in production (duration: 00m 17s) |
[production] |
| 22:30 |
<tgr@tin> |
Synchronized php-1.27.0-wmf.4/includes/specials/SpecialSearch.php: c249899, fixes some warnings in production (duration: 00m 17s) |
[production] |
| 22:29 |
<tgr@tin> |
Synchronized php-1.27.0-wmf.4/includes/Category.php: c249890, fixes some warnings in production (duration: 00m 18s) |
[production] |
| 22:02 |
<gwicke> |
restbase: switched local_group_default_T_parsoid_html to Date-Tiered compaction (DTCS) |
[production] |
| 21:40 |
<tgr@tin> |
Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066 (duration: 00m 18s) |
[production] |
| 21:32 |
<ori@tin> |
Synchronized php-1.27.0-wmf.4/includes/libs/Xhprof.php: (no message) (duration: 00m 18s) |
[production] |
| 21:06 |
<ori@tin> |
Synchronized php-1.27.0-wmf.4/extensions/Cite/Cite_body.php: Revert "Avoid counting arrays if not needed" (duration: 00m 17s) |
[production] |
| 20:56 |
<gwicke> |
restbase: switch local_group_wikiquote_T_title__revisions to Date-Tiered compaction (DTCS) |
[production] |