|
2016-08-24
ยง
|
| 22:49 |
<mutante> |
chromium - revoking and re-signing puppet certs, salt keys, initial puppet run.. |
[production] |
| 22:43 |
<yuvipanda> |
force a puppet run on californium |
[production] |
| 22:27 |
<bd808> |
Deployed striker (2fdf103) to californium |
[production] |
| 22:20 |
<mutante> |
rebooting chromium into PXE |
[production] |
| 22:15 |
<papaul> |
installing puppetmaster200[1-2] |
[production] |
| 22:05 |
<mutante> |
stopping puppet and pdns-recursor on chromium |
[production] |
| 21:57 |
<krinkle@tin> |
Synchronized multiversion/MWMultiVersion.php: Ie9c568a87ae - No-op clean-up (duration: 00m 49s) |
[production] |
| 21:54 |
<yuvipanda> |
forcing puppet run on californium |
[production] |
| 21:50 |
<mutante> |
running puppet on lvs servers, removing chromium from resolv.conf for reinstall |
[production] |
| 21:45 |
<yuvipanda> |
re-enabled puppet, ran puppet and verified ORES is ok on scb1001 / 1002 |
[production] |
| 21:40 |
<yuvipanda> |
re-enabled puppet, ran puppet and verified ORES is ok on scb2001 / 2002 |
[production] |
| 21:27 |
<dzahn@palladium> |
conftool action : set/pooled=no; selector: name=chromium.wikimedia.org |
[production] |
| 21:27 |
<mutante> |
depooling chromium for reinstall. scheduled downtime for host and service IPs |
[production] |
| 21:25 |
<yuvipanda> |
disable puppet on scb2001 and 2002 as well |
[production] |
| 21:24 |
<yuvipanda> |
disable puppet on scb1001 and 1002 |
[production] |
| 21:20 |
<yuvipanda> |
forcing puppet run on tin |
[production] |
| 21:19 |
<yuvipanda> |
forcing puppet run on scb1001 |
[production] |
| 21:14 |
<bd808> |
Forced puppet run on logstash1003 but cron had beaten me to it |
[production] |
| 21:12 |
<bd808> |
Forcing puppet run on logstash1002 |
[production] |
| 21:08 |
<hoo> |
Ran DELETE FROM wbs_propertypairs WHERE pid1 = '641' on Wikidata for T132839 |
[production] |
| 21:07 |
<bd808> |
Forcing puppet run on logstash1001 |
[production] |
| 20:59 |
<urandom> |
T137474: Upgrading xenon.eqiad.wmnet to cassandra_2.2.6-wmf2 |
[production] |
| 20:57 |
<bd808> |
Created initial tables for striker in striker db on m5-master |
[production] |
| 20:53 |
<yuvipanda> |
gave bd808 password for striker db on terbium |
[production] |
| 20:39 |
<yurik> |
deployed kartotherian https://gerrit.wikimedia.org/r/#/c/306518/ |
[production] |
| 20:31 |
<yurik> |
deployed tilerator https://gerrit.wikimedia.org/r/#/c/306514/ |
[production] |
| 20:03 |
<urandom> |
T137474 Starting htmldumper in RESTBase Staging |
[production] |
| 20:02 |
<hashar@tin> |
Synchronized php-1.28.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: Fix hooks signatures T143817 (duration: 00m 47s) |
[production] |
| 19:47 |
<mobrovac> |
change-prop deploying 28a0057 |
[production] |
| 19:00 |
<hashar@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.16 |
[production] |
| 18:44 |
<thcipriani@tin> |
Synchronized wmf-config/CirrusSearch-production.php: SWAT: [[gerrit:306463|Cirrus: Send more like this queries to default cluster]] (duration: 00m 46s) |
[production] |
| 18:37 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:306454|Record content security policy events in log stash]] (duration: 00m 52s) |
[production] |
| 18:23 |
<thcipriani@tin> |
Synchronized wmf-config: SWAT: [[gerrit:306435|Change ORES threshold to soft as default (T143738)]] (duration: 01m 01s) |
[production] |
| 18:17 |
<paravoid> |
reenabling cr2-codfw:xe-5/0/1 (link to cr2-eqiad), recovered since 17:02 UTC |
[production] |
| 18:17 |
<ebernhardson> |
restarted kibana on logstash1001 |
[production] |
| 17:44 |
<jynus> |
overwritting /etc/default/prometheus-mysqld-exporter on all trusty mysql nodes on codfw |
[production] |
| 17:27 |
<gehel> |
deleting logstash indices from before august 1st (T142357) |
[production] |
| 17:04 |
<demon@tin> |
Synchronized wmf-config/: Remove old obsolete ExtensionMessages files (duration: 00m 49s) |
[production] |
| 16:50 |
<godog> |
bounce uwsgi on labmon1001 T143556 |
[production] |
| 16:15 |
<ema@palladium> |
conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe']) |
[production] |
| 16:15 |
<ema@palladium> |
conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx']) |
[production] |
| 15:58 |
<papaul> |
!log wdqs200[1-2] - signing puppet certs, salt-key, initial run |
[production] |
| 15:41 |
<ema@palladium> |
conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe']) |
[production] |
| 15:41 |
<ema@palladium> |
conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx']) |
[production] |
| 15:26 |
<_joe_> |
attempting to restart apache on rhodium, swapping, load exploding |
[production] |
| 15:22 |
<papaul> |
installing wdqs200[1-2] |
[production] |
| 14:41 |
<demon@tin> |
Synchronized multiversion/getMWVersion.php: fixup error message (duration: 00m 47s) |
[production] |
| 13:53 |
<gehel> |
rolling restart of logstash nodes to validate fix to T142357 |
[production] |
| 13:50 |
<hashar> |
European SWAT done |
[production] |
| 13:49 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: Fully restrict uploads on ms.wikipedia T141227 (duration: 00m 46s) |
[production] |