|
2017-01-05
§
|
| 07:54 |
<elukey> |
chown www-data:www-data all the root:adm hhvm log files on mw eqiad hosts (T132324) |
[production] |
| 07:11 |
<marostegui> |
Compressing revision tables across all the wikis - db1038 - T154465 |
[production] |
| 07:09 |
<marostegui> |
Compressing pagelinks tables across all the wikis - db1044 - T153826 |
[production] |
| 07:08 |
<marostegui> |
Compressing revision tables across all the wikis - db1015 - T153739 |
[production] |
| 07:07 |
<marostegui> |
Compressing pagelinks tables across all the wikis - db1044 - T150438 |
[production] |
| 06:15 |
<bd808> |
sudo -u l10nupdate rm /var/lock/scap on tin to clean up lock left by bad l10nupdate locking attempt |
[production] |
| 05:49 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on swift-fe-codfw, swift-be-codfw, ALL remaining mw |
[production] |
| 05:44 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on db-core-codfw, etcd, graphite, kafka-analytics-canary, kafka-analytics, logstash |
[production] |
| 05:39 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on prometheus, aqs, db-es |
[production] |
| 05:35 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on cp-eqiad, memcached-eqiad |
[production] |
| 01:48 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on redis-codfw (rdb2005 needed manual) and all of dc-ulsfo, dc-esams |
[production] |
| 01:41 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on ganeti, cp-ulsfo, wdqs, thumbor, db-es-codfw |
[production] |
| 01:36 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on parsoid, maps, cp-esams |
[production] |
| 01:32 |
<mutante> |
servermon - after the next update by cron - package data is back |
[production] |
| 01:13 |
<mattflaschen@tin> |
Synchronized wmf-config/InitialiseSettings.php: Disable NewUserMessage on gomwiki (duration: 00m 41s) |
[production] |
| 01:10 |
<robh@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw2090.codfw.wmnet |
[production] |
| 01:06 |
<demon@tin> |
Synchronized scap/plugins: (no message) (duration: 00m 40s) |
[production] |
| 01:04 |
<mattflaschen@tin> |
Synchronized php-1.29.0-wmf.7/extensions/Flow: Flow script to add more troubleshooting information to a maintenance script (duration: 00m 56s) |
[production] |
| 00:57 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2090.codfw.wmnet |
[production] |
| 00:57 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2089.codfw.wmnet |
[production] |
| 00:57 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2088.codfw.wmnet |
[production] |
| 00:57 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2087.codfw.wmnet |
[production] |
| 00:57 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2086.codfw.wmnet |
[production] |
| 00:56 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2085.codfw.wmnet |
[production] |
| 00:55 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2084.codfw.wmnet |
[production] |
| 00:55 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2083.codfw.wmnet |
[production] |
| 00:55 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2082.codfw.wmnet |
[production] |
| 00:55 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2081.codfw.wmnet |
[production] |
| 00:54 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2080.codfw.wmnet |
[production] |
| 00:54 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2080.codfw.wmnet |
[production] |
| 00:52 |
<mattflaschen@tin> |
Synchronized php-1.29.0-wmf.6/extensions/Flow: Two Flow fixes related to production database/content inconsistencies. (duration: 00m 59s) |
[production] |
| 00:42 |
<mutante> |
phab2001 - same chmod 751 on exim4 dirs that i manually did on krypton is done by puppet here, fully automatic, not sure why krypton was a one-off |
[production] |
| 00:41 |
<mutante> |
planet1001/2001, phab2001 - upgrade exim4, exim4-daemon-heavy |
[production] |
| 00:37 |
<mutante> |
servermon - weird behaviour in the "pending package upgrades" list? exim4 package was shown as pending on lots of hosts, after next upgrade it disppears from list, even though about half the servers should still be listed |
[production] |
| 00:23 |
<mutante> |
krypton - chmod 751 /var/spool/exim4/ to fix Icinga alerts about unaccesible tmpfs (nagios user could not access), it was 751 on other hosts like ununpentium |
[production] |
| 00:17 |
<mutante> |
krypton - stop exim, umount orphaned "scan" tmpfs (there is no clamav here) |
[production] |
| 00:00 |
<mutante> |
gerrit slowdown reported around 23:55 UTC, was back to normal after 2 minutes (T148478) - attaching latest jvm_gc log |
[production] |
| 00:00 |
<aaron@tin> |
Synchronized wmf-config/logging.php: No-op sync of 7e103f21a3555fc0b8f7fdea4fd8df4cb7cb939e (duration: 00m 42s) |
[production] |
|
2017-01-04
§
|
| 23:55 |
<mutante> |
krypton - chown Debian-exim:Debian-exim /var/spool/exim4/scan/ to fix Icinga-reported DISK issue - wrong permissions - see puppet/modules/exim4/manifests/init.pp line 57 ff "catch-22 with Puppet vs. package" |
[production] |
| 23:20 |
<mutante> |
rolled out exim4 upgrades (DSA 3747-1) on memcached-canary, memcached-codfw, restbase-codfw, cp-codfw |
[production] |
| 23:11 |
<aaron@tin> |
Synchronized wmf-config/logging.php: Include DB shard as a logstash column (duration: 00m 41s) |
[production] |
| 23:09 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on memcached-canary, memcached-codfw, restbase-codfw |
[production] |
| 23:06 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on mw-eqiad |
[production] |
| 22:52 |
<robh> |
all my server depools and decoms for the mw range are on T154621 |
[production] |
| 22:51 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2078.codfw.wmnet |
[production] |
| 22:51 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2077.codfw.wmnet |
[production] |
| 22:51 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2076.codfw.wmnet |
[production] |
| 22:50 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2075.codfw.wmnet |
[production] |
| 22:49 |
<godog> |
rename / reimage restbase-test1* to restbase-dev1* |
[production] |
| 22:46 |
<bblack> |
TLS: unified certificates in esams switching to digicert |
[production] |