|
2018-07-31
ยง
|
| 21:05 |
<twentyafterfour@deploy1001> |
Finished scap: testwikis wikis to 1.32.0-wmf.15 (duration: 44m 49s) |
[production] |
| 20:20 |
<twentyafterfour@deploy1001> |
Started scap: testwikis wikis to 1.32.0-wmf.15 |
[production] |
| 19:00 |
<urandom> |
enabling `unchecked_tombstone_compaction` on enwiki_T_mobile__ng_remaining -- T192689 |
[production] |
| 18:20 |
<XioNoX> |
stopping puppet across the fleet for puppetmaster1001 uplink move |
[production] |
| 18:04 |
<XioNoX> |
stopping pybal on lvs1002 |
[production] |
| 18:01 |
<XioNoX> |
repool lvs1001 |
[production] |
| 17:53 |
<XioNoX> |
stopping pybal on lvs1001 |
[production] |
| 17:34 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=dns,service=pdns_recursor,name=chromium.wikimedia.org |
[production] |
| 17:19 |
<twentyafterfour> |
branching 1.32.0-wmf.15 refs T191061 |
[production] |
| 17:12 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool all hosts in row B - T183585 (duration: 00m 51s) |
[production] |
| 16:40 |
<elukey> |
repool druid1005 after network maintenance |
[production] |
| 16:26 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=dns,service=pdns_recursor,name=chromium.wikimedia.org |
[production] |
| 16:25 |
<godog> |
repool thumbor100[12] - T183585 |
[production] |
| 16:25 |
<elukey> |
pool eventbus on kafka1002 after network maintenance |
[production] |
| 16:24 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=aqs,service=cassandra,name=aqs1004.eqiad.wmnet |
[production] |
| 16:24 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=aqs,service=aqs,name=aqs1004.eqiad.wmnet |
[production] |
| 16:16 |
<elukey> |
precautionary restart of eventbus on kafka1002 after network downtime (DNS name res errors, Kafka broker conn issues, etc..) |
[production] |
| 16:07 |
<marostegui> |
Reload dbproxy1003 and dbproxy1008 |
[production] |
| 16:06 |
<ema> |
reboot cp3044 for kernel update |
[production] |
| 16:01 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=prometheus,service=prometheus,name=prometheus1004.eqiad.wmnet |
[production] |
| 15:51 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=druid-public,service=druid-public-broker,name=druid1005.eqiad.wmnet |
[production] |
| 15:50 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=aqs,service=cassandra,name=aqs1004.eqiad.wmnet |
[production] |
| 15:50 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=aqs,service=aqs,name=aqs1004.eqiad.wmnet |
[production] |
| 15:46 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=eventbus,service=eventbus,name=kafka1002.eqiad.wmnet |
[production] |
| 15:45 |
<ayounsi@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=prometheus,service=prometheus,name=prometheus1004.eqiad.wmnet |
[production] |
| 15:36 |
<ema> |
power-cycle cp3031, stuck rebooting |
[production] |
| 15:29 |
<_joe_> |
restarting apache on phab1001 |
[production] |
| 15:20 |
<godog> |
depool thumbor100[12] ahead of switch move - T183585 |
[production] |
| 15:01 |
<XioNoX> |
starting the eqiad row B servers move - T183585 |
[production] |
| 13:51 |
<moritzm> |
installing ffmpeg 3.2.12 security updates on video scalers |
[production] |
| 13:34 |
<akosiaris> |
repool mathoid eqiad cluster in discovery |
[production] |
| 13:18 |
<akosiaris> |
upgrade kubernetes1004 to 1.9.9 |
[production] |
| 13:14 |
<akosiaris> |
upgrade kubernetes1002 to 1.9.9 |
[production] |
| 13:14 |
<akosiaris> |
upgrade kubernetes1003 to 1.9.9 |
[production] |
| 13:03 |
<akosiaris> |
upgrade kubernetes1001 to 1.9.9 |
[production] |
| 12:58 |
<akosiaris> |
upgrade chlorine.eqiad.wmnet to kubernetes 1.9.9 |
[production] |
| 12:54 |
<akosiaris> |
upgrade argon.eqiad.wmnet to kubernetes 1.9.9 |
[production] |
| 12:45 |
<akosiaris> |
depool eqiad mathoid in discovery |
[production] |
| 12:29 |
<gehel@puppetmaster1001> |
conftool action : set/weight=15; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.eqiad.wmnet |
[production] |
| 12:29 |
<gehel@puppetmaster1001> |
conftool action : set/weight=15; selector: dc=eqiad,cluster=wdqs,name=wdqs1004.eqiad.wmnet |
[production] |
| 12:29 |
<gehel> |
rebalance LVS weights to send less traffic to wdqs1003 - T200563 |
[production] |
| 12:13 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Remove logging now in mw core (duration: 00m 48s) |
[production] |
| 12:11 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: wmg - for Sentry (duration: 00m 48s) |
[production] |
| 12:09 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Remove wmg - for Sentry (duration: 00m 48s) |
[production] |
| 12:06 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Load Sentry with wfLoadExtension (duration: 00m 48s) |
[production] |
| 12:05 |
<reedy@deploy1001> |
Synchronized wmf-config/liquidthreads.php: Remove lqt cruft, load with wfLoadExtension (duration: 00m 48s) |
[production] |
| 11:44 |
<gehel> |
restarting blazegraph on wdqs1003 - JVM unresponsive |
[production] |
| 11:38 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:37 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:448050|Introduce autopatrolled on bnwikisource (T199475)]] (duration: 00m 48s) |
[production] |
| 11:30 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:449391|Use HD logos for hewiktionary (T200470)]] (duration: 00m 48s) |
[production] |