|
2015-06-29
§
|
| 23:57 |
<robh> |
mw2027 was offline (blank screen on serial console). mgmt powercycled |
[production] |
| 23:48 |
<godog> |
start upgrading restbase1* to cassandra 2.1.7 |
[production] |
| 23:41 |
<gwicke> |
restarted cassandra instance on restbase1004.eqiad; log showed many small writes and clients saw timeouts |
[production] |
| 23:29 |
<gwicke> |
deployed restbase 32db4ce1e1 |
[production] |
| 23:21 |
<ori> |
Synchronized php-1.26wmf11/includes/resourceloader: I0e5f2d3b2: resourceloader: Add timing metrics for key operations (duration: 01m 12s) |
[production] |
| 23:15 |
<catrope> |
Synchronized wmf-config/: wikitech cleanup (duration: 01m 08s) |
[production] |
| 23:11 |
<RoanKattouw> |
ssh: connect to host mw2027.codfw.wmnet port 22: Connection timed out |
[production] |
| 23:11 |
<RoanKattouw> |
Synced wmf-config/CommonSettings.php: Remove survey access point in Popups |
[production] |
| 23:09 |
<godog> |
stop ircecho on neon, icinga spam |
[production] |
| 22:53 |
<gwicke> |
canary deploy of restbase 32db4ce1e1 on restbase1001.eqiad |
[production] |
| 21:30 |
<urandom> |
restarting restbase1004 to apply new metrics reporting interval |
[production] |
| 20:19 |
<subbu> |
deployed parsoid sha ea98be88 |
[production] |
| 18:18 |
<ori> |
Synchronized php-1.26wmf11/includes/db/LoadBalancer.php: I0e5f2d3b2: Use APC for caching slave lag times (duration: 01m 09s) |
[production] |
| 18:00 |
<cmjohnson1> |
powering down ms-be1015 |
[production] |
| 16:06 |
<bblack> |
re-enabling puppet on caches |
[production] |
| 15:51 |
<bblack> |
disabling puppet on caches temporarily ... |
[production] |
| 15:49 |
<krenair> |
Synchronized php-1.26wmf11/extensions/OpenStackManager: https://gerrit.wikimedia.org/r/#/c/221648/ (duration: 00m 13s) |
[production] |
| 15:29 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221405/ (duration: 00m 15s) |
[production] |
| 15:27 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221612/ (duration: 00m 12s) |
[production] |
| 15:24 |
<krenair> |
Synchronized w/static/images/project-logos/zhwiki-hans-2x.png: https://gerrit.wikimedia.org/r/#/c/221113/ (duration: 00m 14s) |
[production] |
| 15:24 |
<krenair> |
Synchronized w/static/images/project-logos/zhwiki-hans-1.5x.png: https://gerrit.wikimedia.org/r/#/c/221113/ (duration: 00m 12s) |
[production] |
| 15:23 |
<krenair> |
Synchronized w/static/images/project-logos/zhwiki-hans.png: https://gerrit.wikimedia.org/r/#/c/221113/ (duration: 00m 12s) |
[production] |
| 15:21 |
<krenair> |
Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/221009/ (duration: 00m 11s) |
[production] |
| 15:18 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221047/ (duration: 00m 13s) |
[production] |
| 15:12 |
<krenair> |
Synchronized php-1.26wmf11/extensions/ContentTranslation/modules/tools/ext.cx.tools.link.js: https://gerrit.wikimedia.org/r/#/c/221605 (duration: 00m 13s) |
[production] |
| 15:02 |
<krenair> |
Synchronized php-1.26wmf11/extensions/ContentTranslation/modules/tools/ext.cx.tools.formatter.js: https://gerrit.wikimedia.org/r/#/c/221604/ (duration: 00m 14s) |
[production] |
| 14:34 |
<jynus> |
rebooting and reinstalling db1022 |
[production] |
| 12:06 |
<YuviPanda> |
restarting rsync with new exclusions file on labstore1002 to codfw |
[production] |
| 12:06 |
<YuviPanda> |
excluded maps, mwoffliner and video project from rsync of broken FS to speed it up |
[production] |
| 11:59 |
<YuviPanda> |
interupt rsync on labstore1001 to prevent it from copying mwofflienr files |
[production] |
| 11:00 |
<_joe_> |
shutting down etcd1003, cleaning exported resources |
[production] |
| 10:32 |
<_joe_> |
effectively removing etcd1003 from the cluster |
[production] |
| 10:17 |
<_joe_> |
starting removal of etcd1003 from the etcd cluster |
[production] |
| 08:49 |
<_joe_> |
joined conf1003 to the etcd cluster |
[production] |
| 08:21 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: Depool db1022 for reinstall (duration: 00m 12s) |
[production] |
| 08:12 |
<_joe_> |
adding conf1002 to the etcd cluster as a member |
[production] |
| 07:46 |
<akosiaris> |
disabling ntp everywhere expect selected hosts in anticipation for the leap second |
[production] |
| 04:51 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Jun 29 04:51:48 UTC 2015 (duration 51m 47s) |
[production] |
| 03:08 |
<jgage> |
jmxtrans filled disks on all kafka brokers, 21GB log files. removed logs and restarted services. |
[production] |
| 02:23 |
<LocalisationUpdate> |
completed (1.26wmf11) at 2015-06-29 02:23:47+00:00 |
[production] |
| 02:20 |
<l10nupdate> |
Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 05m 53s) |
[production] |
| 00:52 |
<springle> |
restart eventlogging auto-purge on m4 |
[production] |
| 00:51 |
<springle> |
restart replication on dbstore2002 |
[production] |
| 00:00 |
<springle> |
pausing replication on dbstore2002 |
[production] |