|
2016-05-17
§
|
| 20:00 |
<ejegg> |
disabled banner impression loader |
[production] |
| 19:58 |
<bblack> |
starting upgrade of varnish4 packages on cache_misc |
[production] |
| 19:01 |
<twentyafterfour> |
preparing to deploy wmf/1.28.0-wmf.2 |
[production] |
| 18:37 |
<matt_flaschen> |
Ran manual SQL write in production to work around T122262; see task for query. |
[production] |
| 18:36 |
<urandom> |
Restarting (failed) bootstrap of restbase2008-b.codfw.wmnet : T95253 |
[production] |
| 18:34 |
<urandom> |
Restart restbase2008-a.codfw.wmnet; Hail Mary pass for failed 2008-b bootstraps : T95253 |
[production] |
| 17:22 |
<ejegg> |
updated CiviCRM from 6e3581692bb81c2c507612e7cd66dcdf78ef3cc0 to 7952ba43a012cb6a2e8d16af19bb13ed520bd56f |
[production] |
| 16:23 |
<godog> |
disable mod_deflate and restart apache2 on graphite1001 T135515 |
[production] |
| 16:01 |
<twentyafterfour> |
branching wmf/1.28.0-wmf.2 |
[production] |
| 15:12 |
<thcipriani@tin> |
Synchronized portals: (no message) (duration: 00m 25s) |
[production] |
| 15:12 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 25s) |
[production] |
| 15:11 |
<thcipriani> |
running /srv/mediawiki-staging/portals/sync-portals for [[gerrit:289218]] |
[production] |
| 15:06 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for IP users on the Japanese Wikipedia [[gerrit:287646]] (duration: 00m 35s) |
[production] |
| 14:59 |
<elukey> |
disabled puppet on aqs* nodes as prep step to bootstrap the new testing cluster (https://gerrit.wikimedia.org/r/#/c/288373/5) |
[production] |
| 14:46 |
<gehel> |
taking elastic1001 down for investigation (T135509) |
[production] |
| 14:37 |
<mobrovac> |
change-prop deploying 5b5a07a3 |
[production] |
| 14:15 |
<gehel> |
restarting elastic1001 - high load (T135509) |
[production] |
| 13:32 |
<hashar> |
upgrading Jenkins T133737 |
[production] |
| 13:24 |
<godog> |
bounce carbon/frontend-relay on graphite1001 to increase queue size T135385 |
[production] |
| 13:22 |
<elukey> |
memcahced restarted on mc1009 with -o slab_reassign,slab_automove,lru_crawler,lru_maintainer as part of a perf experiment (T129963) |
[production] |
| 12:21 |
<gehel> |
starting rolling restart of Elasticsearch equiad fro Java update (T135499) |
[production] |
| 12:16 |
<mobrovac> |
mathoid deployed 10c7cb8 |
[production] |
| 11:04 |
<elukey> |
updated facts on puppet compiler following https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet3-diffs |
[production] |
| 10:59 |
<jynus> |
disabling puppet and starting decom process of db1027 |
[production] |
| 10:30 |
<jynus> |
running schema change on s1 T130692 |
[production] |
| 09:59 |
<filippo@palladium> |
conftool action : set/pooled=yes; selector: mw2050.codfw.wmnet |
[production] |
| 09:14 |
<ema> |
upgrading ulsfo cache_misc to varnish 4 (T131501, T134989) |
[production] |
| 09:02 |
<ema> |
upgrading esams cache_misc to varnish 4 (T126206, T134989) |
[production] |
| 08:54 |
<jynus> |
increased pool stall limit to 500 on db1049 |
[production] |
| 08:53 |
<filippo@palladium> |
conftool action : set/pooled=no; selector: mw2050.codfw.wmnet |
[production] |
| 08:31 |
<ema> |
upgrading codfw cache_misc to varnish 4 (T126206, T134989) |
[production] |
| 08:15 |
<jynus> |
reducing durability and enabling GTID on db1026 T135100 |
[production] |
| 08:14 |
<jynus> |
reducing durability and enabling GTID on db1026 |
[production] |
| 08:13 |
<ema> |
upgrading eqiad cache_misc to varnish 4 (T126206, T134989) |
[production] |
| 08:11 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Set db1026 back as rc node; move roles around (duration: 00m 31s) |
[production] |
| 07:08 |
<_joe_> |
restarted hhvm on mw1255, stuck in a deadlock on HPHP::Treadmill::getAgeOldestRequest |
[production] |
| 07:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Reduce db1026 load (duration: 00m 37s) |
[production] |
| 03:44 |
<ori> |
Upgraded Grafana from 3.0.0-pre1 to 3.0.2 |
[production] |
| 02:37 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue May 17 02:37:35 UTC 2016 (duration 8m 46s) |
[production] |
| 02:28 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.1) (duration: 10m 33s) |
[production] |
| 01:13 |
<krenair@tin> |
Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/289118/ (duration: 00m 27s) |
[production] |
| 00:54 |
<YuviPanda> |
restarted rabbitmq on labcontrol1001 |
[production] |
| 00:52 |
<YuviPanda> |
restarted nova-scheduler, let's see if this fixes nodepool |
[production] |
| 00:52 |
<YuviPanda> |
restarted nova-conductor a few mins ago, no help for nodepool |
[production] |
| 00:08 |
<ebernhardson@tin> |
Finished scap: Full scap to sync out WikimediaMessages update (duration: 25m 31s) |
[production] |
| 00:04 |
<mutante> |
created planet2001 ganeti VM on ganeti2001 |
[production] |
| 00:03 |
<mutante> |
mw1230, restart hhvm |
[production] |