|
2013-08-25
§
|
| 16:49 |
<RoanKattouw> |
Restarted Parsoid on a couple of boxes to clear some disk space. It appears wtp1001-1004 have been set up with larger, LVMed root partitions but I can't find any logs on when this was done, by whom, or why |
[production] |
| 16:44 |
<RoanKattouw> |
Restarted Parsoid on wtp1012, was unresponsive |
[production] |
| 16:40 |
<RoanKattouw> |
Strike that, my account on fenari is just being weird apparently |
[production] |
| 16:39 |
<RoanKattouw> |
Investigating low disk space warnings on wtp1001-1024, but it looks like something weird happened to them, I get host key verification warnings or failures for all of them |
[production] |
| 02:37 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sun Aug 25 02:37:03 UTC 2013 |
[production] |
| 02:25 |
<LocalisationUpdate> |
completed (1.22wmf14) at Sun Aug 25 02:25:16 UTC 2013 |
[production] |
| 02:13 |
<LocalisationUpdate> |
completed (1.22wmf13) at Sun Aug 25 02:13:19 UTC 2013 |
[production] |
| 01:03 |
<demon> |
Finished syncing Wikimedia installation... : |
[production] |
| 00:52 |
<demon> |
Started syncing Wikimedia installation... : |
[production] |
|
2013-08-23
§
|
| 23:57 |
<mutante> |
removing mailing list Wikipedia-ensino, creating list educacao-br |
[production] |
| 20:57 |
<marc> |
synchronized docroot |
[production] |
| 20:52 |
<ori-l> |
Re-starting eventlogging/consumer NAME=mysql-db1047 on vanadium |
[production] |
| 20:43 |
<root> |
synchronized docroot |
[production] |
| 20:37 |
<ori-l> |
Stopped eventlogging/consumer NAME=mysql-db1047 on vanadium |
[production] |
| 20:33 |
<binasher> |
taking mysqld down on db1047 (s1 analytics + eventlog data) for unplanned maintenance |
[production] |
| 19:53 |
<RobH> |
ns2.wikimedia.org glue record updated from 91.198.174.4 to 91.198.174.239 |
[production] |
| 18:19 |
<awight> |
civicrm updated from e4f1228d954ec239911143 to 5c28984d7e46c95484 |
[production] |
| 17:18 |
<ottomata> |
disabling puppet on analytics 1006, 1008, 1009, 1021 and 1022 in preparation for reinstallation and repaving next week |
[production] |
| 17:18 |
<awight> |
updated civicrm from 257686e81cf6 to e4f1228d954ec23991 |
[production] |
| 16:27 |
<reedy> |
synchronized php-1.22wmf14/extensions/WikiEditor/ |
[production] |
| 16:25 |
<reedy> |
synchronized php-1.22wmf13/extensions/WikiEditor/ |
[production] |
| 16:19 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 15:44 |
<ottomata> |
disabling puppet on analytics1024 and analytics1025 in preperation for reinstall of zookeeper nodes |
[production] |
| 15:06 |
<ottomata> |
taking analytics1023 offline for reinstall |
[production] |
| 10:15 |
<paravoid> |
upgrading ceph to v0.67.2 |
[production] |
| 07:27 |
<springle> |
enabled dns reverse-lookups on db1016 mysql while replication catches up, to avoid slave sql thread stopping on SET PASSWORD / GRANT statements using hostnames |
[production] |
| 06:08 |
<springle> |
ran mysql_upgrade on db1001 (due to missing/incorrect system tables errors in log) |
[production] |
| 05:24 |
<springle> |
db35 manual mysql.* check, restart, mysql_upgrade, resolve replication sql error |
[production] |
| 05:18 |
<springle> |
stopping db35 mysqld for mysql.* checks and mysql_upgrade |
[production] |
| 05:12 |
<springle> |
resolved db1016 replication sql error |
[production] |
| 02:59 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Aug 23 02:59:10 UTC 2013 |
[production] |
| 02:47 |
<LocalisationUpdate> |
completed (1.22wmf14) at Fri Aug 23 02:47:36 UTC 2013 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.22wmf13) at Fri Aug 23 02:15:37 UTC 2013 |
[production] |
| 01:19 |
<reedy> |
synchronized php-1.22wmf14/includes/specials/SpecialUserlogin.php |
[production] |