|
2014-11-20
§
|
| 15:52 |
<_joe_> |
disabling puppet on all caches, before a pretty large change, will be reeanbled after a few tests |
[production] |
| 15:01 |
<hashar> |
Restarting Jenkins AND Zuul. Beta cluster jobs are still deadlocked. |
[production] |
| 14:00 |
<godog> |
restart txstatsd on tungsten to stop receiving jenkins metrics |
[production] |
| 13:04 |
<hashar> |
Jenkins: restarting to remove a deadlock and unload the Statsd plugin |
[production] |
| 12:30 |
<godog> |
upload carbon-c-relay to trusty-wikimedia |
[production] |
| 12:12 |
<qchris> |
Restarted EventLogging mysql-m2 consumer to empty its caches |
[production] |
| 05:41 |
<bblack> |
amssq31-62, cp300[12], lvs300[34], ssl300[123] all shut down for esams power event (and downtimed) |
[production] |
| 05:28 |
<bblack> |
~30m to esams power out, starting equipment shutdown and such for OE13/OE15 |
[production] |
| 05:15 |
<Tim> |
made myself an administrator on phabricator |
[production] |
| 04:27 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Nov 20 04:27:25 UTC 2014 (duration 27m 24s) |
[production] |
| 02:34 |
<LocalisationUpdate> |
completed (1.25wmf9) at 2014-11-20 02:34:12+00:00 |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.25wmf8) at 2014-11-20 02:21:35+00:00 |
[production] |
| 00:50 |
<maxsem> |
Synchronized php-1.25wmf9/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/174613/ (duration: 00m 04s) |
[production] |
| 00:41 |
<catrope> |
Synchronized wmf-config/InitialiseSettings.php: Change mobile wordmark image to relative URL (duration: 00m 04s) |
[production] |
| 00:34 |
<catrope> |
Synchronized php-1.25wmf9/extensions/VisualEditor: SWAT (duration: 00m 04s) |
[production] |
| 00:34 |
<catrope> |
Synchronized php-1.25wmf8/extensions/VisualEditor: SWAT (duration: 00m 04s) |
[production] |
| 00:26 |
<catrope> |
Synchronized php-1.25wmf8/includes/media/: SWAT: don't apply EXIF rotation to chained thumbnails (duration: 00m 04s) |
[production] |
| 00:13 |
<catrope> |
Synchronized wmf-config/: SWAT: temp debugging for SecurePoll (duration: 00m 04s) |
[production] |
| 00:10 |
<catrope> |
Synchronized wmf-config/: SWAT (duration: 00m 04s) |
[production] |
| 00:09 |
<catrope> |
Synchronized images/mobile/: SWAT: new Wikipedia wordmark for mobile (duration: 00m 03s) |
[production] |
|
2014-11-19
§
|
| 22:01 |
<ottomata> |
starting trusty upgrade of analytics1041 |
[production] |
| 21:38 |
<yuvipandajs> |
restarted txstatsd & carbon on labmon1001, recovering from missing points now |
[production] |
| 21:35 |
<reedy> |
Synchronized wmf-config/: ContactPage for legal (duration: 00m 17s) |
[production] |
| 21:34 |
<reedy> |
Synchronized docroot and w: (no message) (duration: 00m 15s) |
[production] |
| 21:25 |
<ottomata> |
starting trusty upgrade of analytics1040 |
[production] |
| 21:22 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf9 |
[production] |
| 21:21 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.25wmf8 |
[production] |
| 21:18 |
<reedy> |
Finished scap: testwiki to 1.25wmf9 and build l10n cache (duration: 105m 06s) |
[production] |
| 20:57 |
<ottomata> |
starting trusty upgrade of analytics1038 |
[production] |
| 20:23 |
<ottomata> |
starting trusty upgrade of analytics1038 |
[production] |
| 19:49 |
<bblack> |
starting the long slow process of draining out esams traffic ahead of power maint event |
[production] |
| 19:34 |
<ottomata> |
starting trusty upgrade of analytics1037 |
[production] |
| 19:33 |
<reedy> |
Started scap: testwiki to 1.25wmf9 and build l10n cache |
[production] |
| 19:30 |
<andrewbogott> |
upgrading other compute hosts: virt1001-1009 |
[production] |
| 19:25 |
<ori> |
disarming keyholder agent on tin to test alerts |
[production] |
| 18:53 |
<reedy> |
Synchronized php-1.25wmf8/extensions/Wikidata: Ie105a80aa776769eb0dae8a44cda0b7dbe018fb5 (duration: 00m 22s) |
[production] |
| 18:50 |
<andrewbogott> |
upgrading virt1006 |
[production] |
| 18:36 |
<andrewbogott> |
upgrading labnet1001 |
[production] |
| 18:33 |
<andrewbogott> |
upgraded glance on virt1000 to version icehouse |
[production] |
| 18:29 |
<ottomata> |
starting trusty upgrade of 1036 |
[production] |
| 18:18 |
<reedy> |
Synchronized wmf-config/: BounceHandler (duration: 00m 15s) |
[production] |
| 18:17 |
<reedy> |
Synchronized php-1.25wmf8/extensions/BounceHandler/: Bump (duration: 00m 14s) |
[production] |
| 17:57 |
<andrewbogott> |
disabled keystone-redis because the current package doesn't work with icehouse |
[production] |
| 16:36 |
<andrewbogott> |
upgrading virt1000 |
[production] |
| 16:16 |
<andrewbogott> |
moved virt1000 db backup to /a/osback because it was /way/ too big to fit in my homedir |
[production] |
| 16:12 |
<hashar> |
Jenkins: uninstalled Jenkins statsd plugin ( https://phabricator.wikimedia.org/T1278 ). It is overloading the statsd server with a bunch of metrics we don't care about ( https://phabricator.wikimedia.org/T1075 ) |
[production] |
| 15:57 |
<andrewbogott> |
backed up all labs openstack databases to virt1000:~andrew/osback/havana-db-backup.sql |
[production] |
| 15:49 |
<andrewbogott> |
backing up labs configs in ~andrew/osback/<servicename> |
[production] |
| 15:48 |
<andrewbogott> |
beginning upgrade of labs OpenStack from Havana to Icehouse |
[production] |
| 09:44 |
<hoo> |
Synchronized php-1.25wmf8/extensions/Wikidata/: Fix url and commonsMedia UI editing (duration: 00m 42s) |
[production] |