|
2017-01-05
§
|
| 00:42 |
<mutante> |
phab2001 - same chmod 751 on exim4 dirs that i manually did on krypton is done by puppet here, fully automatic, not sure why krypton was a one-off |
[production] |
| 00:41 |
<mutante> |
planet1001/2001, phab2001 - upgrade exim4, exim4-daemon-heavy |
[production] |
| 00:37 |
<mutante> |
servermon - weird behaviour in the "pending package upgrades" list? exim4 package was shown as pending on lots of hosts, after next upgrade it disppears from list, even though about half the servers should still be listed |
[production] |
| 00:23 |
<mutante> |
krypton - chmod 751 /var/spool/exim4/ to fix Icinga alerts about unaccesible tmpfs (nagios user could not access), it was 751 on other hosts like ununpentium |
[production] |
| 00:17 |
<mutante> |
krypton - stop exim, umount orphaned "scan" tmpfs (there is no clamav here) |
[production] |
| 00:00 |
<mutante> |
gerrit slowdown reported around 23:55 UTC, was back to normal after 2 minutes (T148478) - attaching latest jvm_gc log |
[production] |
| 00:00 |
<aaron@tin> |
Synchronized wmf-config/logging.php: No-op sync of 7e103f21a3555fc0b8f7fdea4fd8df4cb7cb939e (duration: 00m 42s) |
[production] |
|
2017-01-04
§
|
| 23:55 |
<mutante> |
krypton - chown Debian-exim:Debian-exim /var/spool/exim4/scan/ to fix Icinga-reported DISK issue - wrong permissions - see puppet/modules/exim4/manifests/init.pp line 57 ff "catch-22 with Puppet vs. package" |
[production] |
| 23:20 |
<mutante> |
rolled out exim4 upgrades (DSA 3747-1) on memcached-canary, memcached-codfw, restbase-codfw, cp-codfw |
[production] |
| 23:11 |
<aaron@tin> |
Synchronized wmf-config/logging.php: Include DB shard as a logstash column (duration: 00m 41s) |
[production] |
| 23:09 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on memcached-canary, memcached-codfw, restbase-codfw |
[production] |
| 23:06 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on mw-eqiad |
[production] |
| 22:52 |
<robh> |
all my server depools and decoms for the mw range are on T154621 |
[production] |
| 22:51 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2078.codfw.wmnet |
[production] |
| 22:51 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2077.codfw.wmnet |
[production] |
| 22:51 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2076.codfw.wmnet |
[production] |
| 22:50 |
<robh@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2075.codfw.wmnet |
[production] |
| 22:49 |
<godog> |
rename / reimage restbase-test1* to restbase-dev1* |
[production] |
| 22:46 |
<bblack> |
TLS: unified certificates in esams switching to digicert |
[production] |
| 22:19 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on mw-codfw |
[production] |
| 21:55 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki nowikinews (T45917) |
[production] |
| 21:55 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki nowiki (T45917) |
[production] |
| 21:48 |
<otto@tin> |
Finished deploy [eventstreams/deploy@a103be2]: (no message) (duration: 01m 32s) |
[production] |
| 21:46 |
<otto@tin> |
Starting deploy [eventstreams/deploy@a103be2]: (no message) |
[production] |
| 21:24 |
<bsitzmann@tin> |
Finished deploy [mobileapps/deploy@c39bd1f]: Update mobileapps to b43c5d6 (duration: 02m 55s) |
[production] |
| 21:21 |
<bsitzmann@tin> |
Starting deploy [mobileapps/deploy@c39bd1f]: Update mobileapps to b43c5d6 |
[production] |
| 21:15 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@3762556]: (no message) (duration: 02m 40s) |
[production] |
| 21:13 |
<smalyshev@tin> |
Starting deploy [wdqs/wdqs@3762556]: (no message) |
[production] |
| 21:01 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@3762556]: (no message) (duration: 00m 59s) |
[production] |
| 21:00 |
<smalyshev@tin> |
Starting deploy [wdqs/wdqs@3762556]: (no message) |
[production] |
| 20:49 |
<madhuvishy> |
adding temporary IP tables rule on labservices1001 to drop traffic from toolchecker for tests (T152369) |
[production] |
| 20:39 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on mw-canary hosts |
[production] |
| 20:36 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on stat* and kubernetes hosts |
[production] |
| 20:05 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.7 |
[production] |
| 19:09 |
<thcipriani@tin> |
Synchronized portals: SWAT: [[gerrit:330450|Bumping portal to master]] T128546 (duration: 00m 42s) |
[production] |
| 19:08 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:330450|Bumping portal to master]] T128546 (duration: 00m 43s) |
[production] |
| 18:30 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on misc servers |
[production] |
| 18:29 |
<ejegg> |
enabled payment processor audit parser jobs |
[production] |
| 17:47 |
<matt_flaschen> |
Ran manual DB update to officewiki for T153320. |
[production] |
| 15:20 |
<zeljkof> |
EU SWAT finished |
[production] |
| 15:19 |
<hashar@tin> |
Synchronized wmf-config/throttle.php: (no message) (duration: 00m 41s) |
[production] |
| 15:03 |
<zeljkof> |
extending eu swat until 330392 is merged |
[production] |
| 14:56 |
<zfilipin@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:330388|Throttle rules for 2017-01-06/07, tewiki (T154568)]] (duration: 00m 40s) |
[production] |
| 14:32 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:277519|Set valid content language for Norwegian wikis (T126146)]] (duration: 00m 41s) |
[production] |
| 14:20 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:329446|Revert $wgMFEEditorOptions[anonymousEditing] = false for kowiki (T119823)]] (duration: 00m 41s) |
[production] |
| 11:18 |
<jynus> |
continuing maintenance on db1035 (mysql replication stopped) |
[production] |
| 10:12 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2060 - T154031 (duration: 00m 47s) |
[production] |
| 07:43 |
<marostegui> |
Compressing tables on db1015 - T153739 |
[production] |
| 07:24 |
<marostegui> |
Compressing more tables on db1044 - T153826 |
[production] |
| 03:10 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Jan 4 03:10:08 UTC 2017 (duration 5m 33s) |
[production] |