|
2016-11-04
§
|
| 22:43 |
<godog> |
stop puppet on einsteinium and tegment to avoid log spam - T150061 |
[production] |
| 21:14 |
<urandom> |
T133395: Starting user-defined compaction of local_group_wikipedia_T_parsoid_html.data, files la-169018-big-Data.db and la-171488-big-Data.db |
[production] |
| 21:06 |
<godog> |
compress huge daemon.log on einsteinium into /srv/ |
[production] |
| 18:11 |
<moritzm> |
uploaded new jessie linux package based on 4.4.30 to carbon |
[production] |
| 18:01 |
<paravoid> |
moving mc1033-mc1036 from asw-d-eqiad to asw2-d-eqiad |
[production] |
| 17:54 |
<paravoid> |
reactivating cr1-eqiad:ae4 and its subinterfaces (VRRP bug seems to have been worked around) |
[production] |
| 17:44 |
<paravoid> |
moved cr1-eqiad:ae4 links from asw-d-eqiad:ae1 to to asw2-d-eqiad:ae1 |
[production] |
| 16:38 |
<ema> |
upgrading cp4018 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 16:22 |
<ema> |
upgrading cp4017 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 16:00 |
<ema> |
upgrading cp4016 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 15:37 |
<ema> |
upgrading cp4010 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 15:36 |
<paravoid> |
set up 4x10G (ae0) links between asw-d-eqiad<->asw2-d-eqiad |
[production] |
| 15:35 |
<marostegui> |
reimage dbstore2002 - T150017 |
[production] |
| 15:20 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove wikitech bot group (duration: 00m 47s) |
[production] |
| 15:17 |
<reedy@tin> |
Synchronized wmf-config/CommonSettings.php: Simplify some wikitech config (duration: 00m 47s) |
[production] |
| 15:16 |
<reedy@tin> |
Synchronized wmf-config/wikitech.php: Stop double loading OATHAuth now, remove commented config (duration: 00m 47s) |
[production] |
| 15:15 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings.php: Normalise wikitech OATHAuth loading config (duration: 00m 48s) |
[production] |
| 15:06 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable OATHAuth on all private wikis (duration: 00m 49s) |
[production] |
| 15:04 |
<reedy@tin> |
Synchronized wmf-config/CommonSettings.php: Raise password requirements for private wikis, Abuse filter editors on enwiki, and make minimum bot password length to 8 (duration: 00m 47s) |
[production] |
| 15:02 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings.php: stage wmgElevateDefaultPasswordPolicy (duration: 00m 48s) |
[production] |
| 14:49 |
<ema> |
upgrading cp4009 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 14:14 |
<ema> |
upgrading cp4008 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 11:33 |
<mobrovac> |
restarting zotero |
[production] |
| 10:58 |
<moritzm> |
installing tar security updates |
[production] |
| 10:21 |
<moritzm> |
upgrading memcached on swift frontend servers in esams |
[production] |
| 10:00 |
<jynus> |
stopping db2011 for backup and reimage |
[production] |
| 09:59 |
<moritzm> |
upgrading memcached on swift frontend servers in codfw |
[production] |
| 09:54 |
<moritzm> |
upgrading memcached on jessie graphite systems |
[production] |
| 09:26 |
<_joe_> |
rebooting copper to allow enabling the memory cgroup |
[production] |
| 09:10 |
<marostegui> |
Reimage db2034 - T149553 |
[production] |
| 07:20 |
<jynus> |
disabling alerting for slave lag fleet-wide for 1 hour to deploy new alerting script |
[production] |
| 06:52 |
<_joe_> |
restarted manually varnish text-backend on cp3041 - failing automatic restarts with "no space left on device" |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Nov 4 02:31:26 UTC 2016 (duration 4m 39s) |
[production] |
| 02:26 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.1) (duration: 09m 08s) |
[production] |
| 01:54 |
<madhuvishy> |
Manually reimaging labstore2003 (T149870) |
[production] |
| 01:35 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.1/extensions/Thanks: Avoid breakage after Flow uninstallation (duration: 00m 47s) |
[production] |
| 01:18 |
<catrope@terbium> |
scap failed: IOError [Errno 13] Permission denied: u'/srv/mediawiki-staging/wmf-config/ExtensionMessages-1.29.0-wmf.1.php' (duration: 00m 20s) |
[production] |
| 01:17 |
<catrope@terbium> |
Started scap: (no message) |
[production] |
| 00:48 |
<catrope@tin> |
Synchronized dblists/: Disable Flow on enwiki (T148611) (duration: 01m 04s) |
[production] |
|
2016-11-03
§
|
| 23:23 |
<thcipriani@tin> |
Synchronized php-1.29.0-wmf.1/extensions/EventBus/EventBus.php: SWAT: [[gerrit:319661|Add logging and check for empty JSON encoded body (T148251)]] (duration: 00m 47s) |
[production] |
| 23:11 |
<thcipriani@tin> |
Synchronized portals: SWAT: [[gerrit:319654|Bumping portals to master (T146807)]] (duration: 00m 52s) |
[production] |
| 23:10 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:319654|Bumping portals to master (T146807)]] (duration: 00m 51s) |
[production] |
| 22:16 |
<ejegg> |
updated CiviCRM from 4ff64afd1ac10643f1f2c91c4aa7a5535512c33e to bdc2786ddaf09e9f412f97406e3cffb13fcc96a |
[production] |
| 20:33 |
<urandom> |
T133395: Enabling unchecked_tombstone_compaction and setting tombstone_threshold = .6 on "local_group_wikipedia_T_parsoid_html".data |
[production] |
| 20:26 |
<bblack> |
codfw cache_text - all nodes v4 and pooled - T131503 |
[production] |
| 20:10 |
<bblack> |
codfw cache_text - all pooled nodes are v4 (2x still depooled-but-upgraded) - T131503 |
[production] |
| 20:04 |
<ejegg> |
enabled donations queue consumer |
[production] |
| 20:02 |
<ejegg> |
disabled donations queue consumer |
[production] |
| 20:01 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.1 |
[production] |
| 19:32 |
<ejegg> |
updated CiviCRM from 954bab421fa3435c5dd1a80f59f657407764b620 to 4ff64afd1ac10643f1f2c91c4aa7a5535512c33e |
[production] |