|
2017-01-31
ยง
|
| 20:32 |
<jynus> |
stopping db1063 mariadb before full host reimage |
[production] |
| 18:58 |
<arlolra> |
Updated Parsoid to version 734dc996 (T98960) |
[production] |
| 18:51 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@dc2323d]: Updating Parsoid to 734dc996 (duration: 12m 58s) |
[production] |
| 18:46 |
<ottomata> |
recentchange events now flowing into Kafka via EventBus T152030 |
[production] |
| 18:45 |
<otto@tin> |
Synchronized wmf-config/CommonSettings.php: Enabling RCFeed -> EventBus (duration: 00m 42s) |
[production] |
| 18:44 |
<otto@tin> |
Synchronized wmf-config/CommonSettings-labs.php: Enabling RCFeed -> EventBus (duration: 00m 43s) |
[production] |
| 18:38 |
<arlolra@tin> |
Started deploy [parsoid/deploy@dc2323d]: Updating Parsoid to 734dc996 |
[production] |
| 18:06 |
<jynus> |
end up tendril and dbtree maintenance, things should be back up, report if you see degradations of service |
[production] |
| 17:37 |
<jynus> |
stopping mysql, upgrading and restarting db1011- temporary outage of tendril & dbtree T111654 |
[production] |
| 16:59 |
<robh> |
disabled puppet on einsteinium while i try to figure out what i broke in my config for icinga |
[production] |
| 16:11 |
<elukey> |
started Cassandra nodetool cleanup for aqs1007-a |
[production] |
| 16:03 |
<elukey> |
started Cassandra nodetool cleanup for aqs1004-b |
[production] |
| 14:57 |
<jynus> |
upgrading and restarting db1095 (sanitarium2) |
[production] |
| 14:12 |
<elukey> |
restarting hhvm on mw1204 (dump debug in /tmp/hhvm.29120.bt) |
[production] |
| 14:07 |
<aude@tin> |
Synchronized wmf-config/Wikibase.php: Update property suggester config (duration: 00m 42s) |
[production] |
| 13:58 |
<elukey> |
rebooted analytics1039 to pick up uuids in fstab - T147879 |
[production] |
| 12:26 |
<addshore> |
TwoColConflict deploy slot done! |
[production] |
| 12:26 |
<addshore@tin> |
Synchronized wmf-config/CommonSettings-labs.php: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 5/5 (duration: 00m 40s) |
[production] |
| 12:25 |
<addshore@tin> |
Synchronized wmf-config/CommonSettings.php: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 4/5 (duration: 00m 40s) |
[production] |
| 12:24 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 3/5 (duration: 00m 42s) |
[production] |
| 12:23 |
<addshore@tin> |
Synchronized wmf-config/extension-list-labs: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 2/5 (duration: 00m 40s) |
[production] |
| 12:22 |
<addshore@tin> |
Synchronized wmf-config/extension-list: [[gerrit:332908|Enable TwoColConflict on test wikis (T155716)]] 1/5 (duration: 00m 40s) |
[production] |
| 12:16 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:332904|Add twocolconflict to wgBetaFeaturesWhitelist (T150184)]] (duration: 00m 41s) |
[production] |
| 11:14 |
<elukey> |
updating the puppet compiler's facts |
[production] |
| 10:42 |
<gehel> |
starting reimage of wdqs1003 |
[production] |
| 09:46 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool hosts in C2 - T155999 (duration: 00m 40s) |
[production] |
| 09:38 |
<moritzm> |
rolling restart of cassandra in codfw to pick up openjdk and NSS security updates |
[production] |
| 09:00 |
<gehel> |
aligning elasticsearch low watermark to 75% disk space on all clusters (eqiad was at 70%) |
[production] |
| 08:44 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=aqs1007.eqiad.wmnet |
[production] |
| 08:40 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Add a warning about a possible bad BBU on db1072 - T156226 (duration: 00m 46s) |
[production] |
| 08:26 |
<elukey> |
started Cassandra nodetool cleanup for aqs1004-a |
[production] |
| 07:54 |
<moritzm> |
installing chromium security update on osmium |
[production] |
| 07:49 |
<marostegui> |
Reboot db1072 to force BBU recharge - T156226 |
[production] |
| 07:10 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2034 - T156478 (duration: 00m 57s) |
[production] |
| 03:49 |
<andrewbogott> |
restarted nova-api on labnet1001 which actually fixed some things |
[production] |
| 03:28 |
<chasemp> |
(slightly belated) set logging level on serpens higher to see if ldap binding is an issue |
[production] |
| 02:45 |
<bd808> |
Setup temporary cron on silver as user bd808 until T156733 is fixed properly |
[production] |
| 02:33 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Jan 31 02:33:22 UTC 2017 (duration 5m 23s) |
[production] |
| 02:31 |
<bd808> |
Manually ran extensions/TorBlock/loadExitNodes.php on silver |
[production] |
| 02:28 |
<bd808@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable TorBlock for Wikitech (duration: 00m 41s) |
[production] |
| 02:28 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.9) (duration: 07m 09s) |
[production] |
| 02:16 |
<chasemp> |
restart uwsgi-keystone-admin and uwsgi-keystone-public on labcontrol1001 |
[production] |
| 00:25 |
<ebernhardson> |
restarting elasticsearch on elastic1029, got stuck in RemoteTransportException loop again |
[production] |
| 00:23 |
<mobrovac@tin> |
Started restart [electron-render/deploy@f1df2d3]: Service restart for firejail upgrade |
[production] |
| 00:22 |
<mobrovac@tin> |
Started restart [mobileapps/deploy@7615bf9]: Service restart for firejail upgrade |
[production] |
| 00:21 |
<mobrovac@tin> |
Started restart [mathoid/deploy@ba3217e]: Service restart for firejail upgrade |
[production] |
| 00:20 |
<mobrovac@tin> |
Started restart [graphoid/deploy@da37386]: Service restart for firejail upgrade |
[production] |
| 00:17 |
<mobrovac@tin> |
Started restart [cxserver/deploy@5ae4f8b]: Service restart for firejail upgrade |
[production] |
| 00:13 |
<mobrovac@tin> |
Started restart [citoid/deploy@95df861]: Service restart for firejail upgrade |
[production] |
| 00:10 |
<mobrovac@tin> |
Started restart [changeprop/deploy@2b980fa]: Service restart for firejail upgrade |
[production] |