|
2017-02-21
§
|
| 12:55 |
<moritzm> |
upgrading openssl on database servers / various base service restarts |
[production] |
| 12:53 |
<volans> |
re-enabled puppet on neodymium and puppetmaster1001 after Gerrit 330436 was merged T154588 |
[production] |
| 12:51 |
<volans> |
re-enabled puppet on planet2001, was disabled since a week without reason |
[production] |
| 12:39 |
<volans> |
reenabled ircecho aftrer fixing ferm issue and run puppet on affected hosts |
[production] |
| 12:08 |
<volans> |
stopped ircecho temporarily while fixing ferm |
[production] |
| 12:01 |
<volans> |
temporarily disabled puppet on neodymium and puppetmaster1001 to merge Gerrit 330436 T154588 |
[production] |
| 11:32 |
<moritzm> |
upgrading openssl on kafka clusters / various base service restarts |
[production] |
| 11:15 |
<moritzm> |
upgrading openssl on restbase clusters / various base service restarts |
[production] |
| 11:05 |
<moritzm> |
upgrading openssl on hadoop cluster / various base service restarts |
[production] |
| 11:02 |
<elukey> |
rolling restart of cassandra-metrics-collector on aqs1* for T157022 |
[production] |
| 10:55 |
<elukey> |
rolling restart of the analyics jmxtrans daemons for T157022 |
[production] |
| 10:29 |
<moritzm> |
restarting base services on mw2* after openssl update |
[production] |
| 10:14 |
<godog> |
downgrade carbon-c-relay on graphite1001 to trusty's version and bounce daemons |
[production] |
| 09:58 |
<moritzm> |
upgrading mira/tin to HHVM 3.12.14 |
[production] |
| 09:46 |
<godog> |
upgrade graphite on graphite1001 and bounce carbon daemons |
[production] |
| 09:26 |
<ema> |
cp3030: libssl1.1 upgraded to 1.1.0e-1+wmf1, libevent-2.0-5 upgraded to 2.0.21-stable-2+deb8u1 |
[production] |
| 08:53 |
<godog> |
switch statsd/graphite DNS to graphite1001 - T157022 |
[production] |
| 08:32 |
<moritzm> |
upgrading mw1170-mw1208 to HHVM 3.12.14 |
[production] |
| 08:30 |
<gehel> |
increasing concurrent recoveries / relocations to 8 on elasticsearch eqiad |
[production] |
| 08:24 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic10(27|32|37|41).eqiad.wmnet |
[production] |
| 07:31 |
<marostegui> |
Deploy alter table enwiki.revision db2055 - T132416 |
[production] |
| 07:29 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2048 and depool db2055 - T132416 (duration: 00m 51s) |
[production] |
| 02:24 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Feb 21 02:24:37 UTC 2017 (duration 5m 20s) |
[production] |
| 02:19 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.12) (duration: 07m 20s) |
[production] |
| 01:17 |
<tstarling@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 42s) |
[production] |
|
2017-02-20
§
|
| 20:31 |
<gehel> |
taking threaddumps and restarting elastic1017 (high load) |
[production] |
| 20:20 |
<gehel> |
reducing concurrent recoveries / relocations to 4 on elasticsearch eqiad |
[production] |
| 19:07 |
<ariel@tin> |
Finished deploy [dumps/dumps@9757356]: fix retries of page content dumps with checkpoint, no dup ranges (duration: 00m 02s) |
[production] |
| 19:07 |
<ariel@tin> |
Started deploy [dumps/dumps@9757356]: fix retries of page content dumps with checkpoint, no dup ranges |
[production] |
| 18:30 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic10(27|32|37|41).eqiad.wmnet |
[production] |
| 18:29 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic10(26|31|36|40).eqiad.wmnet |
[production] |
| 17:56 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@30873eb]: Update change-prop to 30873ebd5: enabling DNS caching for T158338 (duration: 01m 41s) |
[production] |
| 17:54 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@30873eb]: Update change-prop to 30873ebd5: enabling DNS caching for T158338 |
[production] |
| 17:52 |
<Pchelolo> |
update change-prop to 30873ebd5 |
[production] |
| 16:40 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic10(26|31|36|40).eqiad.wmnet |
[production] |
| 14:55 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=wdqs1002.eqiad.wmnet |
[production] |
| 14:49 |
<ema> |
cp2002, cp4008: libssl1.1 upgraded to 1.1.0e-1+wmf1 and libevent-2.0-5 upgraded to 2.0.21-stable-2+deb8u1 |
[production] |
| 14:31 |
<ema> |
upgrading pinkunicorn to varnish 4.1.5-1wm1 |
[production] |
| 14:30 |
<ema> |
varnish 4.1.5-1wm1 uploaded to apt.w.o |
[production] |
| 14:10 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic10(25|28|29|30).eqiad.wmnet |
[production] |
| 13:52 |
<gehel> |
resetting ownership of new .wsp files for wdqs1002 on graphite[12]001 |
[production] |
| 13:49 |
<moritzm> |
installing remaining lcms security updates |
[production] |
| 13:41 |
<hashar@tin> |
Synchronized wmf-config/throttle.php: [throttle] New rule - T158312 (duration: 00m 42s) |
[production] |
| 13:35 |
<marostegui> |
Transferring dbstore1001:/srv/backups (the last 2 backups) to dbstore2001:/srv/backup/dbstore1001 - T153768 |
[production] |
| 13:17 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic10(25|28|29|30).eqiad.wmnet |
[production] |
| 13:04 |
<moritzm> |
installing jasper security updates |
[production] |
| 12:20 |
<godog> |
remove syslog from graphite1001, bump max open files for carbon-c-relay |
[production] |
| 11:00 |
<godog> |
switch diamond traffic to graphite1001 - T157022 |
[production] |
| 10:54 |
<moritzm> |
rolling restart of nginx on remaining mediawiki servers in eqiad to pick up openssl update |
[production] |
| 10:26 |
<ariel@tin> |
Finished deploy [dumps/dumps@dee43ca]: fix prefetch on retries of partially complete page content dumps (duration: 00m 02s) |
[production] |