|
2018-08-31
§
|
| 08:16 |
<godog> |
repair sde1 on ms-be2041 - T199198 |
[production] |
| 07:26 |
<elukey> |
re-run webrequest-load-wf-text-2018-8-31-4, failed due to hadoop workers reboots |
[analytics] |
| 06:41 |
<legoktm> |
deployed https://gerrit.wikimedia.org/r/456547 |
[releng] |
| 06:20 |
<elukey> |
re-run webrequest-load-wf-text-2018-8-31-4, failed for hadoop workers reboots |
[analytics] |
| 05:54 |
<elukey> |
resumed the Hadoop workers reboots for kernel upgrades |
[production] |
| 05:33 |
<elukey> |
restart pdfrender on scb1003 |
[production] |
| 04:26 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/456535 |
[releng] |
| 04:09 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/456534 |
[releng] |
| 03:09 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/456531 |
[releng] |
| 02:42 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/456530 |
[releng] |
| 02:28 |
<legoktm> |
deployed https://gerrit.wikimedia.org/r/456528 |
[releng] |
| 00:44 |
<mutante> |
netmon1002 - restarted smokeping, removed radon as target (unblock decome of former dns server), added cobalt instead as a target also in C4 |
[production] |
|
2018-08-30
§
|
| 23:42 |
<mutante> |
vega - removed rsync config and let puppet regenerate it |
[production] |
| 23:38 |
<mutante> |
bromine - remove outdated rsync conf fragment for static-bugzilla, stopping rsync, running puppet |
[production] |
| 23:36 |
<mutante> |
releases1001 - killing outdated rsync processes for releases from bromine |
[production] |
| 22:57 |
<jforrester@deploy1001> |
Finished scap: Full sync for i18n re-build following Ieaded578ffd (duration: 32m 29s) |
[production] |
| 22:25 |
<jforrester@deploy1001> |
Started scap: Full sync for i18n re-build following Ieaded578ffd |
[production] |
| 22:13 |
<volans> |
this was a test for the logging to IRC, please ignore ^^^^ |
[production] |
| 22:12 |
<END> |
(NOTRUN) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (switchdc/volans@sarin) |
[production] |
| 22:12 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-disable-puppet (switchdc/volans@sarin) |
[production] |
| 21:57 |
<jforrester@deploy1001> |
Synchronized php-1.32.0-wmf.19/extensions/JsonConfig/: Hot-deploy Ieaded578ffd revert of T200968 due to bugs (duration: 00m 51s) |
[production] |
| 20:47 |
<marxarelli> |
Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/455269 for 18 affected jobs |
[releng] |
| 20:34 |
<Hauskatze> |
github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticExpressiveness mirror for T199417 |
[releng] |
| 20:31 |
<Hauskatze> |
github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticLinks mirror for T199422 |
[releng] |
| 20:23 |
<mutante> |
cp1080 - powercycled - lots of RECOVERY from Icinga for IPsec connections - leaving depooled so far (T201174) |
[production] |
| 20:21 |
<Hauskatze> |
github: added standard description to https://github.com/wikimedia/mediawiki-extensions-FundraisingEmailUnsubscribe |
[releng] |
| 20:17 |
<mutante> |
powercycling cp1080 |
[production] |
| 20:06 |
<mutante> |
dzahn@neodymium conftool action : set/pooled=no; selector: name=cp1080.eqiad.wmnet| reason: Strongswan CRITICALs fom Icinga (T201174) |
[production] |
| 20:04 |
<dzahn@neodymium> |
conftool action : set/pooled=no; selector: name=cp1080.eqiad.wmnet |
[production] |
| 19:47 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.19 |
[production] |
| 19:38 |
<thcipriani> |
bring integration-slave-docker-1025 back online |
[releng] |
| 19:37 |
<Hauskatze> |
github: deleted mirror of extension-MwEmbedSupport per T197918 |
[releng] |
| 19:36 |
<marxarelli> |
Deploying 1.32.0-wmf.19 to all wikis |
[production] |
| 19:21 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.19/includes/: I11b390f2e4f5e7 (duration: 01m 16s) |
[production] |
| 19:15 |
<mutante> |
granting project-admin to cwhite |
[puppet] |
| 19:14 |
<mutante> |
adding member cwhite |
[puppet] |
| 19:13 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.19/resources/src/startup: I13a996e01b48 (duration: 01m 06s) |
[production] |
| 18:57 |
<mutante> |
making cwhite a project admin to demo creating instances |
[monitoring] |
| 18:52 |
<moritzm> |
restarting aphlict on phab1001 to pick up nodejs security update |
[production] |
| 18:39 |
<mutante> |
adding new member cwhite |
[monitoring] |
| 16:59 |
<godog> |
xfs_repair on ms-be1041 sdf1 - T199198 (retroactive, started at 15:32 |
[production] |
| 16:35 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.eqiad.wmnet |
[production] |
| 16:30 |
<gehel> |
shutting down wdqs1005 for new SSD and reimaging - T202779 |
[production] |
| 16:29 |
<gehel> |
shutting down wdqs1005 for new SSD and reimaging - T198351 |
[production] |
| 16:15 |
<gehel> |
restart of logstash to move data directory - T198351 |
[production] |
| 16:14 |
<thcipriani> |
reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456362/ |
[releng] |
| 15:59 |
<elukey> |
rerun of pageview-druid-hourly-wf-2018-8-30-13, hadoop worker reboots in progress |
[analytics] |
| 15:46 |
<andrewbogott> |
restarting rabbitmq-server on cloudcontrol1003 |
[admin] |
| 15:23 |
<elukey> |
re-run webrequest-load-wf-upload-2018-8-30-13, failed due to hadoop worker reboots |
[analytics] |
| 15:17 |
<bstorm_> |
increased number of nfsd threads on labstore1004 to 300 |
[production] |