|
2017-02-01
§
|
| 07:13 |
<akosiaris> |
restart thumbor process on thumbor1001, thumbor1002, apply a different LimitNOFILE on thumbo1002 |
[production] |
| 04:17 |
<mutante> |
carbon - rsyncing entire /srv over to install2002 (T156440) |
[production] |
| 03:00 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Feb 1 03:00:32 UTC 2017 (duration 5m 35s) |
[production] |
| 02:54 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.10) (duration: 03m 42s) |
[production] |
| 02:48 |
<mutante> |
install1002, install2002 - install jessie, sign puppet certs, initial puppet run (T132757, T156440) |
[production] |
| 02:34 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.9) (duration: 11m 52s) |
[production] |
| 02:20 |
<demon@tin> |
Synchronized scap/plugins/clean.py: no-op (duration: 00m 39s) |
[production] |
| 01:19 |
<mutante> |
ganeti: create instance install2002 with 80G disk, 2G RAM (T156440) |
[production] |
| 01:15 |
<mutante> |
ganeti: install1001 - remove virtual disk 1 from instance | create instance install1002 instead (T132757) |
[production] |
| 00:57 |
<mutante> |
Ganglia is now deprecated in favor of Grafana (https://phabricator.wikimedia.org/T145659#2925104) |
[production] |
| 00:33 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/334025/ (duration: 00m 40s) |
[production] |
| 00:32 |
<maxsem@tin> |
Synchronized php-1.29.0-wmf.9/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/335263/ (duration: 00m 58s) |
[production] |
|
2017-01-31
§
|
| 23:52 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@e27c3a0]: Update change-prop to fix wikidata rollback rule (duration: 01m 32s) |
[production] |
| 23:51 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@e27c3a0]: Update change-prop to fix wikidata rollback rule |
[production] |
| 22:27 |
<twentyafterfour> |
cleaned up old branches: wmf.3 and wmf.4 |
[production] |
| 21:58 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.29.0-wmf.10 |
[production] |
| 21:50 |
<twentyafterfour@tin> |
Synchronized wmf-config/: sync ExtensionMessages-1.29.0-wmf.10.php (duration: 00m 47s) |
[production] |
| 21:41 |
<twentyafterfour@tin> |
scap sync-l10n completed (1.29.0-wmf.10) (duration: 19m 49s) |
[production] |
| 21:11 |
<twentyafterfour> |
wmf-config/ExtensionMessages-1.29.0-wmf.10.php is missing refs T155525 |
[production] |
| 21:05 |
<twentyafterfour@tin> |
Finished scap: (no justification provided) (duration: 27m 16s) |
[production] |
| 20:37 |
<twentyafterfour@tin> |
Started scap: (no justification provided) |
[production] |
| 20:37 |
<twentyafterfour> |
syncing 1.29.0-wmf.10 to test wikis |
[production] |
| 20:32 |
<jynus> |
stopping db1063 mariadb before full host reimage |
[production] |
| 18:58 |
<arlolra> |
Updated Parsoid to version 734dc996 (T98960) |
[production] |
| 18:51 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@dc2323d]: Updating Parsoid to 734dc996 (duration: 12m 58s) |
[production] |
| 18:46 |
<ottomata> |
recentchange events now flowing into Kafka via EventBus T152030 |
[production] |
| 18:45 |
<otto@tin> |
Synchronized wmf-config/CommonSettings.php: Enabling RCFeed -> EventBus (duration: 00m 42s) |
[production] |
| 18:44 |
<otto@tin> |
Synchronized wmf-config/CommonSettings-labs.php: Enabling RCFeed -> EventBus (duration: 00m 43s) |
[production] |
| 18:38 |
<arlolra@tin> |
Started deploy [parsoid/deploy@dc2323d]: Updating Parsoid to 734dc996 |
[production] |
| 18:06 |
<jynus> |
end up tendril and dbtree maintenance, things should be back up, report if you see degradations of service |
[production] |
| 17:37 |
<jynus> |
stopping mysql, upgrading and restarting db1011- temporary outage of tendril & dbtree T111654 |
[production] |
| 16:59 |
<robh> |
disabled puppet on einsteinium while i try to figure out what i broke in my config for icinga |
[production] |
| 16:11 |
<elukey> |
started Cassandra nodetool cleanup for aqs1007-a |
[production] |
| 16:03 |
<elukey> |
started Cassandra nodetool cleanup for aqs1004-b |
[production] |
| 14:57 |
<jynus> |
upgrading and restarting db1095 (sanitarium2) |
[production] |
| 14:12 |
<elukey> |
restarting hhvm on mw1204 (dump debug in /tmp/hhvm.29120.bt) |
[production] |
| 14:07 |
<aude@tin> |
Synchronized wmf-config/Wikibase.php: Update property suggester config (duration: 00m 42s) |
[production] |
| 13:58 |
<elukey> |
rebooted analytics1039 to pick up uuids in fstab - T147879 |
[production] |
| 12:26 |
<addshore> |
TwoColConflict deploy slot done! |
[production] |
| 12:26 |
<addshore@tin> |
Synchronized wmf-config/CommonSettings-labs.php: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 5/5 (duration: 00m 40s) |
[production] |
| 12:25 |
<addshore@tin> |
Synchronized wmf-config/CommonSettings.php: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 4/5 (duration: 00m 40s) |
[production] |
| 12:24 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 3/5 (duration: 00m 42s) |
[production] |
| 12:23 |
<addshore@tin> |
Synchronized wmf-config/extension-list-labs: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 2/5 (duration: 00m 40s) |
[production] |
| 12:22 |
<addshore@tin> |
Synchronized wmf-config/extension-list: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 1/5 (duration: 00m 40s) |
[production] |
| 12:16 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:332904|Add twocolconflict to wgBetaFeaturesWhitelist (T150184)]] (duration: 00m 41s) |
[production] |
| 11:14 |
<elukey> |
updating the puppet compiler's facts |
[production] |
| 10:42 |
<gehel> |
starting reimage of wdqs1003 |
[production] |
| 09:46 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool hosts in C2 - T155999 (duration: 00m 40s) |
[production] |
| 09:38 |
<moritzm> |
rolling restart of cassandra in codfw to pick up openjdk and NSS security updates |
[production] |
| 09:00 |
<gehel> |
aligning elasticsearch low watermark to 75% disk space on all clusters (eqiad was at 70%) |
[production] |