|
2016-07-20
§
|
| 14:40 |
<Jeff_Green> |
running authdns-update to deploy SPF/DKIM records for wikipedia.org |
[production] |
| 14:20 |
<urandom> |
Restarting bootstrap on restbas1013.eqiad.wmnet (duplicate resume?) : T134016 |
[production] |
| 14:05 |
<urandom> |
Resuming failed bootstrap on restbase1013-c.eqiad.wmnet : T134016 |
[production] |
| 13:29 |
<urandom> |
Performing rolling RESTBase restart to work-around Cassandra instance restart fallout : T138314 and T138314 |
[production] |
| 13:28 |
<urandom> |
Restarting restbase1008-a.eqiad.wmnet to apply a (ephemeral) 7200000ms streaming timeout : T138314 |
[production] |
| 13:15 |
<paravoid> |
cr2-eqiad: setting VRRP priority to 50 for all subnets, effectively switching the VRRP master to cr1-eqiad |
[production] |
| 13:12 |
<_joe_> |
transitioning wtp2011-2020 |
[production] |
| 12:35 |
<_joe_> |
transitioning wtp2002-2010 |
[production] |
| 12:35 |
<oblivian@palladium> |
conftool action : set/pooled=yes; selector: name=wtp2001.codfw.wmnet |
[production] |
| 12:26 |
<_joe_> |
transition ongoing on wtp2001 |
[production] |
| 12:23 |
<oblivian@palladium> |
conftool action : set/pooled=no; selector: name=wtp2001.codfw.wmnet |
[production] |
| 12:22 |
<oblivian@palladium> |
conftool action : set/pooled=no; selector: name= |
[production] |
| 12:17 |
<_joe_> |
disabling puppet on all parsoid hosts for the transition to service-runner T90668 |
[production] |
| 07:28 |
<elukey> |
restarting evenbus on kafka100[12] (T140848) |
[production] |
| 06:14 |
<_joe_> |
updating parsoid on wtp100[12] |
[production] |
| 04:55 |
<mutante> |
osmium package chromium-browser is missing after upgrade, refered to by jsbench |
[production] |
| 04:45 |
<mutante> |
osmium - rsyncing /home , /srv (except /srv/mediawiki created by puppet) back from temp backup on hafnium |
[production] |
| 04:38 |
<mutante> |
osmium result: boots into 4.4 kernel which would not work before.. lol |
[production] |
| 04:35 |
<mutante> |
osmium edit grub config to boot second entry (3.16), update-grub, reboot |
[production] |
| 03:00 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Jul 20 03:00:43 UTC 2016 (duration 6m 50s) |
[production] |
| 02:53 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.11) (duration: 07m 55s) |
[production] |
| 02:29 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 41s) |
[production] |
| 02:28 |
<demon@tin> |
Synchronized wmf-config/CommonSettings.php: delist codereview (duration: 00m 27s) |
[production] |
| 01:39 |
<demon@tin> |
Synchronized tests/SiteConfiguration.php: for completeness (duration: 00m 24s) |
[production] |
| 01:35 |
<mutante> |
hafnium stopping rsyncd, deleting configs |
[production] |
| 01:33 |
<mutante> |
labstore1005, accepting salt key (reinstall 2016-06-25) |
[production] |
| 01:29 |
<mutante> |
rhodium, new puppetmaster, add to salt |
[production] |
| 01:19 |
<mutante> |
osmium - revoke old puppet cert, salt-key .. sign new ones |
[production] |
| 01:19 |
<mutante> |
osmium - after reinstall with jessie, did not boot with 4.4 kernel, _does_ boot with 3.16.04.. still jessie just booted manually into the older kernel in grub |
[production] |
| 01:13 |
<demon@tin> |
Synchronized wmf-config/abusefilter.php: Disable abusefilter profiling on commonswiki (duration: 00m 26s) |
[production] |
| 00:44 |
<dereckson@tin> |
Finished scap: wmf-config/ upgrade: Gerrit changes 296770, 296767, 296929, 296930, 292623, 292624 (duration: 45m 42s) |
[production] |
| 00:42 |
<urandom> |
Temporarily reducing compaction throughput to 10MB/s on restbase1013-c.eqiad.wmnet : T134016 |
[production] |
| 00:32 |
<mutante> |
osmium - reboot into PXE, reinstall |
[production] |
|
2016-07-19
§
|
| 23:58 |
<dereckson@tin> |
Started scap: wmf-config/ upgrade: Gerrit changes 296770, 296767, 296929, 296930, 292623, 292624 |
[production] |
| 22:17 |
<awight> |
update paymentswiki to fundraising/REL1_27 from e8b600c518b28e3f350ced85d7d1006a76b86596 to 7c6fb5a3b90fffdf2229cc903fb546e0e1e47998 |
[production] |
| 22:10 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/299870/ (duration: 00m 29s) |
[production] |
| 21:54 |
<demon@tin> |
Synchronized php-1.28.0-wmf.10/extensions/AbuseFilter: Backported fix for logspam (duration: 00m 38s) |
[production] |
| 21:43 |
<gwicke> |
temporarily lowering compaction throughput on all eqiad restbase cassandra instances from 60mb/s to 20mb/s via `nodetool setcompactionthroughput 20` (T140825) |
[production] |
| 21:23 |
<gwicke> |
temporarily lowered compaction throughput on all 1012 instances from 60mb/s to 20mb/s via `nodetool setcompactionthroughput 20` (T140825) |
[production] |
| 21:08 |
<urandom> |
Bootstrapping restbase2003-c.codfw.wmnet : T134016 |
[production] |
| 20:45 |
<urandom> |
Lowering compaction throughput to 20MB/s on restbase1013-{a,b}.eqiad.wmnet : T134016 |
[production] |
| 20:44 |
<urandom> |
Lowering compaction throughput from 35MB/s to 20MB/s on restbase1013-c.eqiad.wmnet : T134016 |
[production] |
| 20:28 |
<urandom> |
Throttling stream throughput to 20MB/s on all rack 'b' instances : T134016 |
[production] |
| 20:21 |
<urandom> |
Lowering compaction throughput from 45MB/s to 35MB/s on restbase1013-c.eqiad.wmnet : T134016 |
[production] |
| 20:06 |
<urandom> |
Disabling Puppet on restbase2003.codfw.wmnet : T134016 |
[production] |
| 20:03 |
<demon@tin> |
Finished scap: group0 to wmf.11 (duration: 24m 52s) |
[production] |
| 19:57 |
<urandom> |
Reducing stream throughput on restbase1013-{a,b} to 20MB/s : T134016 |
[production] |
| 19:47 |
<urandom> |
Lowering compaction throughput from 60MB/s to 45MB/s on restbase1013-c.eqiad.wmnet : T134016 |
[production] |
| 19:38 |
<demon@tin> |
Started scap: group0 to wmf.11 |
[production] |
| 19:31 |
<demon@tin> |
Purged l10n cache for 1.28.0-wmf.9 |
[production] |