|
2017-02-01
§
|
| 13:38 |
<akosiaris> |
issue sudo hdparm -Y /dev/sdb on bast3001 to force a problematic drive to sleep |
[production] |
| 13:21 |
<marostegui> |
Clean up db1043 replication thread (it was replicating from db1048 which looks like an old thing) - T156905 |
[production] |
| 12:09 |
<elukey@tin> |
Finished deploy [analytics/refinery@e6254a4]: (no justification provided) (duration: 04m 41s) |
[production] |
| 12:04 |
<elukey@tin> |
Started deploy [analytics/refinery@e6254a4]: (no justification provided) |
[production] |
| 11:52 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2061 - T153300 (duration: 00m 40s) |
[production] |
| 11:25 |
<moritzm> |
removing ntfs-3g from various trusty servers |
[production] |
| 11:14 |
<godog> |
bounce leaking thumbor@8813 on thumbor1001 |
[production] |
| 11:08 |
<kartik@tin> |
Finished deploy [cxserver/deploy@0e4ae4f]: (no justification provided) (duration: 02m 04s) |
[production] |
| 11:06 |
<kartik@tin> |
Started deploy [cxserver/deploy@0e4ae4f]: (no justification provided) |
[production] |
| 07:53 |
<marostegui> |
Deploy alter table metawiki.pagelinks db2061 - T153300 |
[production] |
| 07:52 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2061 - T153300 (duration: 00m 53s) |
[production] |
| 07:43 |
<moritzm> |
rolling restart of cassandra in eqiad to pick up openjdk and NSS security updates |
[production] |
| 07:41 |
<elukey> |
bootstrapping aqs1008-a on aqs1008 (new AQS cassandra node) |
[production] |
| 07:31 |
<marostegui> |
Force WB policy on the raid controller db1072 - T156226 |
[production] |
| 07:13 |
<akosiaris> |
restart thumbor process on thumbor1001, thumbor1002, apply a different LimitNOFILE on thumbo1002 |
[production] |
| 07:13 |
<akosiaris> |
restart thumbor process on thumbor1001, thumbor1002, apply a different LimitNOFILE on thumbo1002 |
[production] |
| 04:17 |
<mutante> |
carbon - rsyncing entire /srv over to install2002 (T156440) |
[production] |
| 03:00 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Feb 1 03:00:32 UTC 2017 (duration 5m 35s) |
[production] |
| 02:54 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.10) (duration: 03m 42s) |
[production] |
| 02:48 |
<mutante> |
install1002, install2002 - install jessie, sign puppet certs, initial puppet run (T132757, T156440) |
[production] |
| 02:34 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.9) (duration: 11m 52s) |
[production] |
| 02:20 |
<demon@tin> |
Synchronized scap/plugins/clean.py: no-op (duration: 00m 39s) |
[production] |
| 01:19 |
<mutante> |
ganeti: create instance install2002 with 80G disk, 2G RAM (T156440) |
[production] |
| 01:15 |
<mutante> |
ganeti: install1001 - remove virtual disk 1 from instance | create instance install1002 instead (T132757) |
[production] |
| 00:57 |
<mutante> |
Ganglia is now deprecated in favor of Grafana (https://phabricator.wikimedia.org/T145659#2925104) |
[production] |
| 00:33 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/334025/ (duration: 00m 40s) |
[production] |
| 00:32 |
<maxsem@tin> |
Synchronized php-1.29.0-wmf.9/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/335263/ (duration: 00m 58s) |
[production] |
|
2017-01-31
§
|
| 23:52 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@e27c3a0]: Update change-prop to fix wikidata rollback rule (duration: 01m 32s) |
[production] |
| 23:51 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@e27c3a0]: Update change-prop to fix wikidata rollback rule |
[production] |
| 22:27 |
<twentyafterfour> |
cleaned up old branches: wmf.3 and wmf.4 |
[production] |
| 21:58 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.29.0-wmf.10 |
[production] |
| 21:50 |
<twentyafterfour@tin> |
Synchronized wmf-config/: sync ExtensionMessages-1.29.0-wmf.10.php (duration: 00m 47s) |
[production] |
| 21:41 |
<twentyafterfour@tin> |
scap sync-l10n completed (1.29.0-wmf.10) (duration: 19m 49s) |
[production] |
| 21:11 |
<twentyafterfour> |
wmf-config/ExtensionMessages-1.29.0-wmf.10.php is missing refs T155525 |
[production] |
| 21:05 |
<twentyafterfour@tin> |
Finished scap: (no justification provided) (duration: 27m 16s) |
[production] |
| 20:37 |
<twentyafterfour@tin> |
Started scap: (no justification provided) |
[production] |
| 20:37 |
<twentyafterfour> |
syncing 1.29.0-wmf.10 to test wikis |
[production] |
| 20:32 |
<jynus> |
stopping db1063 mariadb before full host reimage |
[production] |
| 18:58 |
<arlolra> |
Updated Parsoid to version 734dc996 (T98960) |
[production] |
| 18:51 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@dc2323d]: Updating Parsoid to 734dc996 (duration: 12m 58s) |
[production] |
| 18:46 |
<ottomata> |
recentchange events now flowing into Kafka via EventBus T152030 |
[production] |
| 18:45 |
<otto@tin> |
Synchronized wmf-config/CommonSettings.php: Enabling RCFeed -> EventBus (duration: 00m 42s) |
[production] |
| 18:44 |
<otto@tin> |
Synchronized wmf-config/CommonSettings-labs.php: Enabling RCFeed -> EventBus (duration: 00m 43s) |
[production] |
| 18:38 |
<arlolra@tin> |
Started deploy [parsoid/deploy@dc2323d]: Updating Parsoid to 734dc996 |
[production] |
| 18:06 |
<jynus> |
end up tendril and dbtree maintenance, things should be back up, report if you see degradations of service |
[production] |
| 17:37 |
<jynus> |
stopping mysql, upgrading and restarting db1011- temporary outage of tendril & dbtree T111654 |
[production] |
| 16:59 |
<robh> |
disabled puppet on einsteinium while i try to figure out what i broke in my config for icinga |
[production] |
| 16:11 |
<elukey> |
started Cassandra nodetool cleanup for aqs1007-a |
[production] |
| 16:03 |
<elukey> |
started Cassandra nodetool cleanup for aqs1004-b |
[production] |
| 14:57 |
<jynus> |
upgrading and restarting db1095 (sanitarium2) |
[production] |