|
2018-08-31
§
|
| 19:46 |
<mutante> |
right when it was fixed on ms-be2043 it also broke on ms-be2040. following the same instructions to fix xfs in a root screen (T199198) |
[production] |
| 19:12 |
<mutante> |
ms-be2043 - following instructions at https://wikitech.wikimedia.org/wiki/Graphite#Repair_xfs_misreporting_free_space to repair xfs misreporting free space (T199198), fixing docs, icinga-downtime doesn't want fqdn but short name |
[production] |
| 18:10 |
<jforrester@deploy1001> |
Synchronized php-1.32.0-wmf.19/includes/Title.php: Hot-deploy of I05eea553c58 to let users edit [[Copyright]] again (duration: 00m 50s) |
[production] |
| 17:57 |
<mutante> |
depooled wtp2020 because icinga reported memory errors |
[production] |
| 17:39 |
<SMalyshev> |
repooled wdqs1005 |
[production] |
| 16:30 |
<jforrester@deploy1001> |
Synchronized php-1.32.0-wmf.19/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: Hot-deploy of I38eda4aac48 to fix T203213 (duration: 00m 54s) |
[production] |
| 10:49 |
<moritzm> |
installing libgd2 security updates on trusty |
[production] |
| 09:46 |
<moritzm> |
installing libx11 security updates on trusty |
[production] |
| 08:57 |
<gehel> |
elasticsearch data directory migration on all logstash nodes |
[production] |
| 08:16 |
<godog> |
repair sde1 on ms-be2041 - T199198 |
[production] |
| 05:54 |
<elukey> |
resumed the Hadoop workers reboots for kernel upgrades |
[production] |
| 05:33 |
<elukey> |
restart pdfrender on scb1003 |
[production] |
| 00:44 |
<mutante> |
netmon1002 - restarted smokeping, removed radon as target (unblock decome of former dns server), added cobalt instead as a target also in C4 |
[production] |
|
2018-08-30
§
|
| 23:42 |
<mutante> |
vega - removed rsync config and let puppet regenerate it |
[production] |
| 23:38 |
<mutante> |
bromine - remove outdated rsync conf fragment for static-bugzilla, stopping rsync, running puppet |
[production] |
| 23:36 |
<mutante> |
releases1001 - killing outdated rsync processes for releases from bromine |
[production] |
| 22:57 |
<jforrester@deploy1001> |
Finished scap: Full sync for i18n re-build following Ieaded578ffd (duration: 32m 29s) |
[production] |
| 22:25 |
<jforrester@deploy1001> |
Started scap: Full sync for i18n re-build following Ieaded578ffd |
[production] |
| 22:13 |
<volans> |
this was a test for the logging to IRC, please ignore ^^^^ |
[production] |
| 22:12 |
<END> |
(NOTRUN) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (switchdc/volans@sarin) |
[production] |
| 22:12 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-disable-puppet (switchdc/volans@sarin) |
[production] |
| 21:57 |
<jforrester@deploy1001> |
Synchronized php-1.32.0-wmf.19/extensions/JsonConfig/: Hot-deploy Ieaded578ffd revert of T200968 due to bugs (duration: 00m 51s) |
[production] |
| 20:23 |
<mutante> |
cp1080 - powercycled - lots of RECOVERY from Icinga for IPsec connections - leaving depooled so far (T201174) |
[production] |
| 20:17 |
<mutante> |
powercycling cp1080 |
[production] |
| 20:06 |
<mutante> |
dzahn@neodymium conftool action : set/pooled=no; selector: name=cp1080.eqiad.wmnet| reason: Strongswan CRITICALs fom Icinga (T201174) |
[production] |
| 20:04 |
<dzahn@neodymium> |
conftool action : set/pooled=no; selector: name=cp1080.eqiad.wmnet |
[production] |
| 19:47 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.19 |
[production] |
| 19:36 |
<marxarelli> |
Deploying 1.32.0-wmf.19 to all wikis |
[production] |
| 19:21 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.19/includes/: I11b390f2e4f5e7 (duration: 01m 16s) |
[production] |
| 19:13 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.19/resources/src/startup: I13a996e01b48 (duration: 01m 06s) |
[production] |
| 18:52 |
<moritzm> |
restarting aphlict on phab1001 to pick up nodejs security update |
[production] |
| 16:59 |
<godog> |
xfs_repair on ms-be1041 sdf1 - T199198 (retroactive, started at 15:32 |
[production] |
| 16:35 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.eqiad.wmnet |
[production] |
| 16:30 |
<gehel> |
shutting down wdqs1005 for new SSD and reimaging - T202779 |
[production] |
| 16:29 |
<gehel> |
shutting down wdqs1005 for new SSD and reimaging - T198351 |
[production] |
| 16:15 |
<gehel> |
restart of logstash to move data directory - T198351 |
[production] |
| 15:17 |
<bstorm_> |
increased number of nfsd threads on labstore1004 to 300 |
[production] |
| 14:08 |
<moritzm> |
upgrading mw1262-1265 to wikidiff 1.7.3 |
[production] |
| 13:46 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1119, db1090 (s2 and s7) (duration: 00m 59s) |
[production] |
| 13:41 |
<moritzm> |
upgrading mw1261 to wikidiff 1.7.3 |
[production] |
| 13:31 |
<jynus> |
depooling labsdb1009 due to extra lag |
[production] |
| 13:10 |
<jynus> |
sanitizing fixcopyrightwiki on db2094 and children T202820 |
[production] |
| 13:08 |
<jynus> |
sanitizing fixcopyrightwiki on db1124 and children T202820 |
[production] |
| 13:03 |
<moritzm> |
upgrading grafana to 4.6.4 (security release) |
[production] |
| 12:59 |
<elukey> |
drain + reboot analytics10[28-79]* for kernel updates (will take multiple days) |
[production] |
| 12:28 |
<godog> |
roll restart thumbor in eqiad to upgrade to 2.1 |
[production] |
| 12:25 |
<reedy@deploy1001> |
Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 36s) |
[production] |
| 12:23 |
<reedy@deploy1001> |
Synchronized dblists/: fixcopyrightwiki (duration: 00m 56s) |
[production] |
| 12:22 |
<moritzm> |
uploaded grafana 4.6.4 to apt.wikimedia.org |
[production] |
| 12:13 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: fixcopyrightwiki (duration: 00m 56s) |
[production] |