|
2016-08-24
ยง
|
| 21:27 |
<mutante> |
depooling chromium for reinstall. scheduled downtime for host and service IPs |
[production] |
| 21:25 |
<yuvipanda> |
disable puppet on scb2001 and 2002 as well |
[production] |
| 21:24 |
<yuvipanda> |
disable puppet on scb1001 and 1002 |
[production] |
| 21:20 |
<yuvipanda> |
forcing puppet run on tin |
[production] |
| 21:19 |
<yuvipanda> |
forcing puppet run on scb1001 |
[production] |
| 21:14 |
<bd808> |
Forced puppet run on logstash1003 but cron had beaten me to it |
[production] |
| 21:12 |
<bd808> |
Forcing puppet run on logstash1002 |
[production] |
| 21:08 |
<hoo> |
Ran DELETE FROM wbs_propertypairs WHERE pid1 = '641' on Wikidata for T132839 |
[production] |
| 21:07 |
<bd808> |
Forcing puppet run on logstash1001 |
[production] |
| 20:59 |
<urandom> |
T137474: Upgrading xenon.eqiad.wmnet to cassandra_2.2.6-wmf2 |
[production] |
| 20:57 |
<bd808> |
Created initial tables for striker in striker db on m5-master |
[production] |
| 20:53 |
<yuvipanda> |
gave bd808 password for striker db on terbium |
[production] |
| 20:39 |
<yurik> |
deployed kartotherian https://gerrit.wikimedia.org/r/#/c/306518/ |
[production] |
| 20:31 |
<yurik> |
deployed tilerator https://gerrit.wikimedia.org/r/#/c/306514/ |
[production] |
| 20:03 |
<urandom> |
T137474 Starting htmldumper in RESTBase Staging |
[production] |
| 20:02 |
<hashar@tin> |
Synchronized php-1.28.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: Fix hooks signatures T143817 (duration: 00m 47s) |
[production] |
| 19:47 |
<mobrovac> |
change-prop deploying 28a0057 |
[production] |
| 19:00 |
<hashar@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.16 |
[production] |
| 18:44 |
<thcipriani@tin> |
Synchronized wmf-config/CirrusSearch-production.php: SWAT: [[gerrit:306463|Cirrus: Send more like this queries to default cluster]] (duration: 00m 46s) |
[production] |
| 18:37 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:306454|Record content security policy events in log stash]] (duration: 00m 52s) |
[production] |
| 18:23 |
<thcipriani@tin> |
Synchronized wmf-config: SWAT: [[gerrit:306435|Change ORES threshold to soft as default (T143738)]] (duration: 01m 01s) |
[production] |
| 18:17 |
<paravoid> |
reenabling cr2-codfw:xe-5/0/1 (link to cr2-eqiad), recovered since 17:02 UTC |
[production] |
| 18:17 |
<ebernhardson> |
restarted kibana on logstash1001 |
[production] |
| 17:44 |
<jynus> |
overwritting /etc/default/prometheus-mysqld-exporter on all trusty mysql nodes on codfw |
[production] |
| 17:27 |
<gehel> |
deleting logstash indices from before august 1st (T142357) |
[production] |
| 17:04 |
<demon@tin> |
Synchronized wmf-config/: Remove old obsolete ExtensionMessages files (duration: 00m 49s) |
[production] |
| 16:50 |
<godog> |
bounce uwsgi on labmon1001 T143556 |
[production] |
| 16:15 |
<ema@palladium> |
conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe']) |
[production] |
| 16:15 |
<ema@palladium> |
conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx']) |
[production] |
| 15:58 |
<papaul> |
!log wdqs200[1-2] - signing puppet certs, salt-key, initial run |
[production] |
| 15:41 |
<ema@palladium> |
conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe']) |
[production] |
| 15:41 |
<ema@palladium> |
conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx']) |
[production] |
| 15:26 |
<_joe_> |
attempting to restart apache on rhodium, swapping, load exploding |
[production] |
| 15:22 |
<papaul> |
installing wdqs200[1-2] |
[production] |
| 14:41 |
<demon@tin> |
Synchronized multiversion/getMWVersion.php: fixup error message (duration: 00m 47s) |
[production] |
| 13:53 |
<gehel> |
rolling restart of logstash nodes to validate fix to T142357 |
[production] |
| 13:50 |
<hashar> |
European SWAT done |
[production] |
| 13:49 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: Fully restrict uploads on ms.wikipedia T141227 (duration: 00m 46s) |
[production] |
| 13:48 |
<zfilipin@tin> |
Synchronized dblists/commonsuploads.dblist: Fully restrict uploads on ms.wikipedia T141227 (duration: 00m 46s) |
[production] |
| 13:39 |
<ema> |
upgrading cp4005 to Varnish 4 (T131502) |
[production] |
| 13:27 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: Make default ORES threshold soft (higher threshold) T143738 (duration: 00m 59s) |
[production] |
| 12:56 |
<mobrovac> |
changeprop deploying c5fd932 |
[production] |
| 12:29 |
<gehel> |
remove old logstash indices that were not deleted after 31 days |
[production] |
| 11:10 |
<godog> |
enable collection of mysqld metrics from prometheus2002 too |
[production] |
| 10:49 |
<moritzm> |
start installing hhvm updates in eqiad |
[production] |
| 10:13 |
<gehel> |
increase recovery throttling on elasticsearch codfw to reduce rolling restart time |
[production] |
| 09:52 |
<moritzm> |
disabled creation of unprivileged user namespaces on trusty systems via sysctl kernel.unprivileged_userns_clone |
[production] |
| 09:30 |
<gehel> |
starting rolling restart of elasticearch codfw for JVM and elasticsearch upgrade |
[production] |
| 09:03 |
<hashar> |
gallium contint::firewall: Limited to production networks https://gerrit.wikimedia.org/r/301627 . For monitoring do: grep iptables-dropped /var/log/syslog |
[production] |
| 08:39 |
<ema> |
dns: re-enabling codfw |
[production] |