|
2015-01-08
§
|
| 23:46 |
<ori> |
Synchronized wmf-config/mc.php: (no message) (duration: 00m 07s) |
[production] |
| 23:32 |
<ori> |
Synchronized wmf-config/mc.php: Revert: I4c4691e26: memcached: use a unix socket instead of a tcp connection on selected hosts (duration: 00m 06s) |
[production] |
| 23:30 |
<ori> |
Synchronized wmf-config/mc.php: I4c4691e26: memcached: use a unix socket instead of a tcp connection on selected hosts (duration: 00m 06s) |
[production] |
| 23:26 |
<ori> |
depooling mw1230 and mw1231 for a couple of minutes for I4c4691e26 |
[production] |
| 21:53 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 06s) |
[production] |
| 21:52 |
<mutante> |
fixing scap permissions on mediawiki-installation servers via dsh |
[production] |
| 21:29 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 06s) |
[production] |
| 21:04 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 09s) |
[production] |
| 21:00 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: nooop to test scap update (duration: 00m 06s) |
[production] |
| 20:39 |
<Reedy> |
Scap deployed at a78ddec |
[production] |
| 20:22 |
<Reedy> |
moved /srv/deployment/scap to scap.old as git repo seems busted. Hoping puppet puts it back again correctly... |
[production] |
| 19:47 |
<Reedy> |
scap-rebuild-cdbs finished |
[production] |
| 19:44 |
<Reedy> |
running dsh -g mediawiki-installation -M -F 40 -- "sudo -u mwdeploy /srv/deployment/scap/scap/bin/scap-rebuild-cdbs" |
[production] |
| 19:43 |
<reedy> |
Synchronized php-1.25wmf14/cache/l10n/: l10nupdate (duration: 03m 39s) |
[production] |
| 19:39 |
<reedy> |
Synchronized php-1.25wmf13/cache/l10n/: (no message) (duration: 00m 05s) |
[production] |
| 19:38 |
<reedy> |
Synchronized php-1.25wmf13/cache/l10n/: l10nupdate (duration: 03m 54s) |
[production] |
| 19:26 |
<LocalisationUpdate> |
completed (1.25wmf14) at 2015-01-08 19:26:46+00:00 |
[production] |
| 19:25 |
<reedy> |
Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 19:14 |
<LocalisationUpdate> |
completed (1.25wmf13) at 2015-01-08 19:14:37+00:00 |
[production] |
| 19:13 |
<reedy> |
Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 18:58 |
<Reedy> |
Attempting manual run of l10nupdate |
[production] |
| 17:08 |
<marktraceur> |
Finished scap: [SWAT] [AbuseFilter] Add file_size variable (duration: 33m 27s) |
[production] |
| 16:51 |
<hashar> |
Restarted Zuul. Same issue as https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul |
[production] |
| 16:35 |
<marktraceur> |
Started scap: [SWAT] [AbuseFilter] Add file_size variable |
[production] |
| 16:25 |
<marktraceur> |
Synchronized php-1.25wmf14/extensions/AbuseFilter: [SWAT] [AbuseFilter] Add file_size variable (duration: 00m 06s) |
[production] |
| 16:14 |
<marktraceur> |
Synchronized php-1.25wmf14/extensions/VisualEditor/: [SWAT] VisualEditor fixes for T86046 and T86056 (duration: 00m 05s) |
[production] |
| 16:02 |
<marktraceur> |
Synchronized wmf-config/logging.php: [SWAT] Honor log sampling and levels for logstash on group0 wikis (duration: 00m 05s) |
[production] |
| 15:44 |
<_joe_> |
restarting hhvm on mw1226, TC full |
[production] |
| 15:37 |
<_joe_> |
restarting hhvm on mw1113, stuck in parsing the ini file (HPHP::is_valid_var_name) |
[production] |
| 15:08 |
<aude> |
Synchronized wmf-config/Wikibase.php: Bump cache epoch for test.wikidata (duration: 00m 06s) |
[production] |
| 15:01 |
<aude> |
Finished scap: Update group0 to wmf/1.25wmf14 Wikidata extension branch (duration: 26m 43s) |
[production] |
| 14:35 |
<aude> |
Started scap: Update group0 to wmf/1.25wmf14 Wikidata extension branch |
[production] |
| 14:34 |
<Jeff_Green> |
samarium package updates and reboot |
[production] |
| 11:37 |
<godog> |
reboot ms-be1011, xfs hosed :( |
[production] |
| 11:07 |
<_joe_> |
repooled the canary servers |
[production] |
| 10:55 |
<_joe_> |
installing a new hhvm package (with the correct libicu dependence) on canary hosts |
[production] |
| 10:02 |
<godog> |
removing backend hosts from LVS for search pools |
[production] |
| 09:42 |
<godog> |
restart uwsgi on tungsten |
[production] |
| 08:15 |
<_joe_> |
depooled the canary appservers while a new package version is rebuilt |
[production] |
| 08:11 |
<_joe_> |
installing a new hhvm package version on the canary pools |
[production] |
| 03:06 |
<springle> |
xtrabackup clone db1037 to db1050 |
[production] |
| 01:04 |
<bd808> |
cleaned up logstash indices dated 2014-01-* and 2015-12-* that look to have been created by some sort of syslog input parsing bug |
[production] |
| 01:01 |
<bd808> |
accidentally deleted 2015-01-07 logstash index when cleaning up rogue indices for 2014-01-* |
[production] |
| 00:40 |
<bd808> |
restarted elasticsearch on logstash1002 to heal split brain in cluster |
[production] |
| 00:38 |
<bd808> |
elasticsearch cluster for logstash is split brain. |
[production] |
| 00:38 |
<maxsem> |
Synchronized php-1.25wmf13/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/#/c/183186/ (duration: 00m 07s) |
[production] |
| 00:37 |
<maxsem> |
Synchronized php-1.25wmf14/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/#/c/183186/ (duration: 00m 08s) |
[production] |
| 00:34 |
<bd808> |
restarted logstash on logstash1001 |
[production] |
| 00:11 |
<K4-713> |
updated payments settings |
[production] |