|
2016-11-04
§
|
| 10:21 |
<moritzm> |
upgrading memcached on swift frontend servers in esams |
[production] |
| 10:00 |
<jynus> |
stopping db2011 for backup and reimage |
[production] |
| 09:59 |
<moritzm> |
upgrading memcached on swift frontend servers in codfw |
[production] |
| 09:54 |
<moritzm> |
upgrading memcached on jessie graphite systems |
[production] |
| 09:26 |
<_joe_> |
rebooting copper to allow enabling the memory cgroup |
[production] |
| 09:10 |
<marostegui> |
Reimage db2034 - T149553 |
[production] |
| 07:20 |
<jynus> |
disabling alerting for slave lag fleet-wide for 1 hour to deploy new alerting script |
[production] |
| 06:52 |
<_joe_> |
restarted manually varnish text-backend on cp3041 - failing automatic restarts with "no space left on device" |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Nov 4 02:31:26 UTC 2016 (duration 4m 39s) |
[production] |
| 02:26 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.1) (duration: 09m 08s) |
[production] |
| 01:54 |
<madhuvishy> |
Manually reimaging labstore2003 (T149870) |
[production] |
| 01:35 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.1/extensions/Thanks: Avoid breakage after Flow uninstallation (duration: 00m 47s) |
[production] |
| 01:18 |
<catrope@terbium> |
scap failed: IOError [Errno 13] Permission denied: u'/srv/mediawiki-staging/wmf-config/ExtensionMessages-1.29.0-wmf.1.php' (duration: 00m 20s) |
[production] |
| 01:17 |
<catrope@terbium> |
Started scap: (no message) |
[production] |
| 00:48 |
<catrope@tin> |
Synchronized dblists/: Disable Flow on enwiki (T148611) (duration: 01m 04s) |
[production] |
|
2016-11-03
§
|
| 23:23 |
<thcipriani@tin> |
Synchronized php-1.29.0-wmf.1/extensions/EventBus/EventBus.php: SWAT: [[gerrit:319661|Add logging and check for empty JSON encoded body (T148251)]] (duration: 00m 47s) |
[production] |
| 23:11 |
<thcipriani@tin> |
Synchronized portals: SWAT: [[gerrit:319654|Bumping portals to master (T146807)]] (duration: 00m 52s) |
[production] |
| 23:10 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:319654|Bumping portals to master (T146807)]] (duration: 00m 51s) |
[production] |
| 22:16 |
<ejegg> |
updated CiviCRM from 4ff64afd1ac10643f1f2c91c4aa7a5535512c33e to bdc2786ddaf09e9f412f97406e3cffb13fcc96a |
[production] |
| 20:33 |
<urandom> |
T133395: Enabling unchecked_tombstone_compaction and setting tombstone_threshold = .6 on "local_group_wikipedia_T_parsoid_html".data |
[production] |
| 20:26 |
<bblack> |
codfw cache_text - all nodes v4 and pooled - T131503 |
[production] |
| 20:10 |
<bblack> |
codfw cache_text - all pooled nodes are v4 (2x still depooled-but-upgraded) - T131503 |
[production] |
| 20:04 |
<ejegg> |
enabled donations queue consumer |
[production] |
| 20:02 |
<ejegg> |
disabled donations queue consumer |
[production] |
| 20:01 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.1 |
[production] |
| 19:32 |
<ejegg> |
updated CiviCRM from 954bab421fa3435c5dd1a80f59f657407764b620 to 4ff64afd1ac10643f1f2c91c4aa7a5535512c33e |
[production] |
| 19:31 |
<mobrovac> |
change-prop deploying f107669 |
[production] |
| 19:28 |
<twentyafterfour@tin> |
Synchronized php-1.29.0-wmf.1/extensions/PageImages/includes/ApiQueryPageImages.php: T149849 (duration: 00m 47s) |
[production] |
| 19:23 |
<mobrovac> |
restbase restarting to re-include wikidata domains for T149114 |
[production] |
| 19:17 |
<Krenair> |
<twentyafterfour> !log In order to unblock the train for group2: deploying https://gerrit.wikimedia.org/r/#/c/319643/ refs T149059, T149849 |
[production] |
| 18:46 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:319644|Fix comment refs (T148327)]] (duration: 00m 47s) |
[production] |
| 18:39 |
<thcipriani@tin> |
Synchronized wmf-config: SWAT: [[gerrit:319605|LABS: Enable Map (GeoJSON) data on Commons (T149548)]] (housekeeping only sync) (duration: 00m 50s) |
[production] |
| 18:33 |
<thcipriani@tin> |
Synchronized php-1.29.0-wmf.1/extensions/Kartographer/includes/ApiQueryMapData.php: SWAT: [[gerrit:319622|Fix warning (T149923)]] (duration: 00m 47s) |
[production] |
| 18:26 |
<ema> |
repooling cp2016 (T131503) |
[production] |
| 18:21 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.23/extensions/EventBus/EventBus.php: SWAT: [[gerrit:319586|Log more EventBus HTTP request/response context for HTTP errors (T148251)]] (duration: 00m 52s) |
[production] |
| 18:19 |
<thcipriani@tin> |
Synchronized php-1.29.0-wmf.1/extensions/EventBus/EventBus.php: SWAT: [[gerrit:319587|Log more EventBus HTTP request/response context for HTTP errors (T148251)]] (duration: 00m 49s) |
[production] |
| 18:02 |
<hashar> |
Added security rule for "puppet3-diffs" labs project to allow ssh connection from contint1001 instead of gallium |
[production] |
| 16:20 |
<jynus> |
stopping and upgrading labsdb1009,10,11 (also disabling temporarily puppet) |
[production] |
| 16:16 |
<papaul> |
OS install on labstore2001 |
[production] |
| 16:13 |
<mutante> |
mw1205 - service hhvm restart |
[production] |
| 15:59 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: mariadb: Reduce db1051 load, it has hardware issues (duration: 00m 47s) |
[production] |
| 15:41 |
<moritzm> |
installing memcached security updates on graphite hosts |
[production] |
| 15:39 |
<mobrovac> |
scb in eqiad enabled puppet back |
[production] |
| 15:26 |
<mobrovac> |
scb in eqiad disabling puppet |
[production] |
| 14:57 |
<akosiaris> |
failover icinga from tegmen to einsteinium |
[production] |
| 14:17 |
<moritzm> |
exim reenabled on fermium after mailman update |
[production] |
| 14:14 |
<moritzm> |
temporarily stop exim on fermium for mailman update |
[production] |
| 14:08 |
<bblack> |
cache_text: nginx lossless restarts for libssl update - T144626 - T148917 |
[production] |
| 14:00 |
<mobrovac> |
restbase rolling restart for T149114 |
[production] |
| 13:56 |
<bblack> |
cache_upload: nginx lossless restarts for libssl update - T144626 - T148917 |
[production] |