|
2015-05-19
§
|
| 21:07 |
<robh> |
merging fixes to sodium, mailing list outage fixed |
[production] |
| 20:51 |
<andrewbogott> |
rebooting/reimaging virt1005, virt1006, 1007 |
[production] |
| 20:22 |
<mutante> |
mailman: killed processes by user "list". started mailman |
[production] |
| 19:40 |
<ori> |
Synchronized wmf-config/CommonSettings.php: Ia6a2cb7: Removed "refreshLinks" from $wgJobBackoffThrottling (duration: 00m 12s) |
[production] |
| 19:37 |
<anomie> |
Finished scap: Step 2 for deploying ApiFeatureUsage: sync the config, and l10n data again because I don't think it did last time (duration: 44m 34s) |
[production] |
| 19:25 |
<robh> |
mailman permission errors abound! had to take it offline again and fixing |
[production] |
| 19:02 |
<robh> |
mailman is back to routing mail normally (still testing rename parts) |
[production] |
| 18:53 |
<anomie> |
Started scap: Step 2 for deploying ApiFeatureUsage: sync the config, and l10n data again because I don't think it did last time |
[production] |
| 18:51 |
<anomie> |
Finished scap: Step 1 for deploying ApiFeatureUsage: sync the code and l10n data (duration: 05m 39s) |
[production] |
| 18:46 |
<anomie> |
Started scap: Step 1 for deploying ApiFeatureUsage: sync the code and l10n data |
[production] |
| 18:39 |
<yuvipanda> |
issuing start command for all hosts on labvirt1006, just to make sure |
[production] |
| 18:37 |
<yuvipanda> |
labvirt1006 rebooting, long POST |
[production] |
| 18:32 |
<yuvipanda> |
restarted labvirt1006 |
[production] |
| 18:21 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.26wmf6 |
[production] |
| 18:17 |
<robh> |
stopping mailman again for further planned work T99098 |
[production] |
| 17:44 |
<robh> |
mailing lists still down, scrubbing list archives is painful and error prone |
[production] |
| 17:34 |
<ottomata> |
starting reboots of analytics worker nodes in order to enable hyperthreading Bug: https://phabricator.wikimedia.org/T90640 |
[production] |
| 17:06 |
<robh> |
puppet stopped on sodium (dont need it restarting mailman while im working) |
[production] |
| 17:05 |
<robh> |
starting mailman downtime window to scrub content off list archive per T99098 |
[production] |
| 17:00 |
<bblack> |
automated reboots of esams/eqiad non-upload caches starting (should auto-downtime, should be no real impact)... |
[production] |
| 15:51 |
<anomie> |
Synchronized php-1.26wmf5/extensions/AbuseFilter/: SWAT: Fix boolean response in API action=abusefiltercheckmatch [[gerrit:211743]] (duration: 00m 12s) |
[production] |
| 15:50 |
<anomie> |
Synchronized php-1.26wmf6/extensions/AbuseFilter/: SWAT: Fix boolean response in API action=abusefiltercheckmatch [[gerrit:211744]] (duration: 00m 10s) |
[production] |
| 15:31 |
<anomie> |
Synchronized php-1.26wmf5/includes/skins/SkinTemplate.php: SWAT: Revert "output mw-content-{ltr,rtl} unconditionally" [[gerrit:211893]] (duration: 00m 12s) |
[production] |
| 15:28 |
<anomie> |
Synchronized php-1.26wmf6/includes/skins/SkinTemplate.php: SWAT: Revert "output mw-content-{ltr,rtl} unconditionally" [[gerrit:211894]] (duration: 00m 13s) |
[production] |
| 15:16 |
<anomie> |
Synchronized php-1.26wmf5/includes/registration/ExtensionRegistry.php: SWAT: registration: Don't array_unique() over the queue before loading it [[gerrit:211948] (duration: 00m 12s) |
[production] |
| 15:15 |
<anomie> |
Synchronized php-1.26wmf6/includes/registration/ExtensionRegistry.php: SWAT: registration: Don't array_unique() over the queue before loading it [[gerrit:211947] (duration: 00m 12s) |
[production] |
| 14:43 |
<jynus> |
back to read/write after virt1000 database migration - migration seems ok |
[production] |
| 14:41 |
<godog> |
purge cassandra system CF metrics from graphite1001 |
[production] |
| 14:29 |
<jynus> |
temporarily going read-only for virt1000 for database migration |
[production] |
| 14:24 |
<mobrovac> |
enabled puppet on restbase1001 |
[production] |
| 14:19 |
<mobrovac> |
restbase group1 wiki keyspaces created |
[production] |
| 14:15 |
<mobrovac> |
starting manually RB with group1 wikis enabled on restbase1001 |
[production] |
| 14:11 |
<mobrovac> |
restbase100x: removed superfluous keyspaces by hand from Cassandra |
[production] |
| 13:49 |
<Steinsplitter> |
test |
[production] |
| 13:47 |
<bblack> |
done with cp40xx reboot process |
[production] |
| 13:32 |
<bblack> |
rebooting ulsfo caches (cp40xx - currently depooled from all traffic + downtimed in icinga) |
[production] |
| 13:09 |
<mobrovac> |
disabled puppet on restbase100x |
[production] |
| 12:51 |
<godog> |
bounce hhvm on mw1152 |
[production] |
| 08:26 |
<_joe_> |
restarting a few HHVM instances with a full TC space |
[production] |
| 05:05 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue May 19 05:03:56 UTC 2015 (duration 3m 55s) |
[production] |
| 02:46 |
<LocalisationUpdate> |
completed (1.26wmf6) at 2015-05-19 02:45:17+00:00 |
[production] |
| 02:43 |
<krinkle> |
Synchronized php-1.26wmf6/includes/resourceloader/ResourceLoader.php: Ic0df4fb5cff (duration: 00m 12s) |
[production] |
| 02:42 |
<l10nupdate> |
Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 43s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.26wmf5) at 2015-05-19 02:25:05+00:00 |
[production] |
| 02:21 |
<l10nupdate> |
Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 11s) |
[production] |
| 00:37 |
<ebernhardson> |
Synchronized php-1.26wmf5/includes/jobqueue/JobQueueGroup.php: Undefer push() in lazyPush() temporarily (duration: 00m 12s) |
[production] |
| 00:36 |
<ebernhardson> |
Synchronized php-1.26wmf6/includes/jobqueue/JobQueueGroup.php: Undefer push() in lazyPush() temporarily (duration: 00m 12s) |
[production] |