|
2017-02-27
§
|
| 12:39 |
<elukey> |
restart zookeeper on conf2002 |
[production] |
| 12:14 |
<_joe_> |
reissuing the certificate for etcd.codfw.wmnet due to a previous error |
[production] |
| 12:00 |
<elukey> |
rebooting mw2092 due to puppet errors for mw-cgroup - T151427 |
[production] |
| 11:58 |
<volans> |
re-enabled icinga-wm |
[production] |
| 11:37 |
<ema> |
cp1052 repooled T148891 |
[production] |
| 11:19 |
<elukey> |
zookeeper status report - new changes rolled out to druid nodes and conf2001 - conf1* and conf200[23] still pending, waiting for more metrics before proceeding |
[production] |
| 11:09 |
<volans> |
temporarily stopped ircecho (icinga-wm) |
[production] |
| 11:04 |
<ema> |
rebooting cp1052 into kernel 4.4.2-3+wmf8 T148891 |
[production] |
| 10:49 |
<moritzm> |
uploaded apache2 2.4.10-10+deb8u8+wmf1 to apt.wikimedia.org (rebase of local patches on top on latest DSA) |
[production] |
| 10:34 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: wme: Set ReadingDepth sampling rate to 0.1% - T155639 (duration: 00m 40s) |
[production] |
| 10:31 |
<elukey> |
limiting the Zookeeper Maximum heap size to 1G (https://gerrit.wikimedia.org/r/#/c/337797/) - setting applied gradually to Zookeeper on Druid and Conf* hosts |
[production] |
| 10:11 |
<_joe_> |
upgrading conftool to 0.4.0 across the cluster T149617 |
[production] |
| 10:03 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1045 for maintenance (duration: 00m 43s) |
[production] |
| 09:19 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1026 after maintenance with full weight (duration: 00m 39s) |
[production] |
| 08:42 |
<_joe_> |
upload conftool 0.4.0 to trusty-wikimedia |
[production] |
| 08:42 |
<_joe_> |
promote conftool 0.4.0 to jessie-wikimedia main |
[production] |
| 07:59 |
<marostegui> |
Run pt-table-checksum on s2 (nlwiki) on revision table - T154485 |
[production] |
| 07:29 |
<marostegui> |
Deploy alter table enwiki.revision - db2034 - T132416 |
[production] |
| 07:21 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2034 - T132416 (duration: 00m 40s) |
[production] |
| 07:15 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2070 - T132416 (duration: 00m 40s) |
[production] |
| 02:25 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Feb 27 02:25:10 UTC 2017 (duration 5m 21s) |
[production] |
| 02:19 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.13) (duration: 07m 23s) |
[production] |
|
2017-02-24
§
|
| 22:46 |
<Krinkle> |
(terbium) sql --write mediawikiwiki 'DELETE FROM module_deps' (in batches of 500; 42292 rows affected) - per T158105. |
[production] |
| 22:28 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@62354ed]: Deploy new updater on 1001 for timeout increase (duration: 00m 16s) |
[production] |
| 22:27 |
<smalyshev@tin> |
Started deploy [wdqs/wdqs@62354ed]: Deploy new updater on 1001 for timeout increase |
[production] |
| 22:23 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@62354ed]: Deploy new updater on 2001 for testing (duration: 00m 26s) |
[production] |
| 22:23 |
<smalyshev@tin> |
Started deploy [wdqs/wdqs@62354ed]: Deploy new updater on 2001 for testing |
[production] |
| 20:50 |
<ebernhardson> |
restart elasticsearch on logstash1002 |
[production] |
| 20:05 |
<demon@tin> |
Synchronized wmf-config/wikitech.php: (no justification provided) (duration: 00m 48s) |
[production] |
| 19:30 |
<Pchelolo> |
restarting RESTBase on xenon.eqiad.wmnet in staging |
[production] |
| 17:15 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1026 after maintenance with low load (duration: 00m 40s) |
[production] |
| 16:55 |
<volans> |
manually cleaning ferm leftovers on dbproxy1011 - T158798 |
[production] |
| 15:35 |
<ema> |
temporarily bumping timeout_idle to 60s on cache_misc T154558 |
[production] |
| 14:27 |
<volans> |
re-started and re-armed keyholder after upgrade on: mira.codfw.wmnet,neodymium.eqiad.wmnet,sarin.codfw.wmnet,tin.eqiad.wmnet T158660 T158659 |
[production] |
| 10:41 |
<ema> |
cache_misc: upgrading to varnish 4.1.5 |
[production] |
| 10:30 |
<moritzm> |
installing imagemagick regression update for security update on trusty (the Debian update seems unaffected) |
[production] |
| 10:23 |
<moritzm> |
installing spice updates on trusty |
[production] |
| 09:48 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1036 - T154485 (duration: 00m 40s) |
[production] |
| 09:39 |
<elukey> |
stop Redis and Memcached on mc2001->mc2016 as extra precautionary step before decom - T157675 |
[production] |
| 08:44 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet |
[production] |
| 08:16 |
<volans> |
temporary disabled puppet on neodymium and sarin to deploy Gerrit 339183 - T158753 |
[production] |
| 07:32 |
<marostegui> |
Deploy alter table enwiki.revision on db2070 - T132416 |
[production] |