|
2014-11-19
§
|
| 21:38 |
<yuvipandajs> |
restarted txstatsd & carbon on labmon1001, recovering from missing points now |
[production] |
| 21:35 |
<reedy> |
Synchronized wmf-config/: ContactPage for legal (duration: 00m 17s) |
[production] |
| 21:34 |
<reedy> |
Synchronized docroot and w: (no message) (duration: 00m 15s) |
[production] |
| 21:25 |
<ottomata> |
starting trusty upgrade of analytics1040 |
[production] |
| 21:22 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf9 |
[production] |
| 21:21 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.25wmf8 |
[production] |
| 21:18 |
<reedy> |
Finished scap: testwiki to 1.25wmf9 and build l10n cache (duration: 105m 06s) |
[production] |
| 20:57 |
<ottomata> |
starting trusty upgrade of analytics1038 |
[production] |
| 20:23 |
<ottomata> |
starting trusty upgrade of analytics1038 |
[production] |
| 19:49 |
<bblack> |
starting the long slow process of draining out esams traffic ahead of power maint event |
[production] |
| 19:34 |
<ottomata> |
starting trusty upgrade of analytics1037 |
[production] |
| 19:33 |
<reedy> |
Started scap: testwiki to 1.25wmf9 and build l10n cache |
[production] |
| 19:30 |
<andrewbogott> |
upgrading other compute hosts: virt1001-1009 |
[production] |
| 19:25 |
<ori> |
disarming keyholder agent on tin to test alerts |
[production] |
| 18:53 |
<reedy> |
Synchronized php-1.25wmf8/extensions/Wikidata: Ie105a80aa776769eb0dae8a44cda0b7dbe018fb5 (duration: 00m 22s) |
[production] |
| 18:50 |
<andrewbogott> |
upgrading virt1006 |
[production] |
| 18:36 |
<andrewbogott> |
upgrading labnet1001 |
[production] |
| 18:33 |
<andrewbogott> |
upgraded glance on virt1000 to version icehouse |
[production] |
| 18:29 |
<ottomata> |
starting trusty upgrade of 1036 |
[production] |
| 18:18 |
<reedy> |
Synchronized wmf-config/: BounceHandler (duration: 00m 15s) |
[production] |
| 18:17 |
<reedy> |
Synchronized php-1.25wmf8/extensions/BounceHandler/: Bump (duration: 00m 14s) |
[production] |
| 17:57 |
<andrewbogott> |
disabled keystone-redis because the current package doesn't work with icehouse |
[production] |
| 16:36 |
<andrewbogott> |
upgrading virt1000 |
[production] |
| 16:16 |
<andrewbogott> |
moved virt1000 db backup to /a/osback because it was /way/ too big to fit in my homedir |
[production] |
| 16:12 |
<hashar> |
Jenkins: uninstalled Jenkins statsd plugin ( https://phabricator.wikimedia.org/T1278 ). It is overloading the statsd server with a bunch of metrics we don't care about ( https://phabricator.wikimedia.org/T1075 ) |
[production] |
| 15:57 |
<andrewbogott> |
backed up all labs openstack databases to virt1000:~andrew/osback/havana-db-backup.sql |
[production] |
| 15:49 |
<andrewbogott> |
backing up labs configs in ~andrew/osback/<servicename> |
[production] |
| 15:48 |
<andrewbogott> |
beginning upgrade of labs OpenStack from Havana to Icehouse |
[production] |
| 09:44 |
<hoo> |
Synchronized php-1.25wmf8/extensions/Wikidata/: Fix url and commonsMedia UI editing (duration: 00m 42s) |
[production] |
| 05:44 |
<legoktm> |
batchCAAntiSpoof finished with "34721605 user(s) done." |
[production] |
| 04:57 |
<ejegg> |
updated tools from 419fb7aa32c6d0776056968378e358ee01985565 to fe9b463379fac35ad5e71a57fbbb95ae39e2356e |
[production] |
| 04:42 |
<Tim> |
on mw1114: testing xhprof hack for T758 |
[production] |
| 04:15 |
<ori> |
mw1020: disabled puppet & restarted hhvm w/hhvm.eval.perf_pid_map = true to test |
[production] |
| 04:15 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Nov 19 04:15:00 UTC 2014 (duration 14m 59s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.25wmf8) at 2014-11-19 02:26:12+00:00 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.25wmf7) at 2014-11-19 02:14:12+00:00 |
[production] |
| 01:28 |
<qchris> |
Restarted EventLogging mysql-m2 consumer to pick up switch to dbproxy1002 |
[production] |
| 01:23 |
<ori> |
Synchronized php-1.25wmf7/extensions/SyntaxHighlight_GeSHi: I788e1beb8: Update SyntaxHighlight_GeSHi for cherry-picks (duration: 00m 05s) |
[production] |
| 01:20 |
<ori> |
Synchronized php-1.25wmf8/extensions/SyntaxHighlight_GeSHi: Ibb0f7c24: Update SyntaxHighlight_GeSHi for cherry-picks (duration: 00m 05s) |
[production] |
| 00:48 |
<springle> |
m2-master CNAME switch to dbproxy1002, and db1046 to primary backend |
[production] |
| 00:38 |
<ori> |
EventLogging: deployed 423f7dd5b2b5 & restarted. |
[production] |
| 00:21 |
<maxsem> |
Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/174303 (duration: 00m 05s) |
[production] |
| 00:09 |
<MaxSem> |
gracefulled apache on mw1205 (suspect an APC bug) |
[production] |
| 00:06 |
<ori> |
repooled mw1205 |
[production] |