|
2016-08-24
§
|
| 18:23 |
<thcipriani@tin> |
Synchronized wmf-config: SWAT: [[gerrit:306435|Change ORES threshold to soft as default (T143738)]] (duration: 01m 01s) |
[production] |
| 18:17 |
<paravoid> |
reenabling cr2-codfw:xe-5/0/1 (link to cr2-eqiad), recovered since 17:02 UTC |
[production] |
| 18:17 |
<ebernhardson> |
restarted kibana on logstash1001 |
[production] |
| 17:44 |
<jynus> |
overwritting /etc/default/prometheus-mysqld-exporter on all trusty mysql nodes on codfw |
[production] |
| 17:27 |
<gehel> |
deleting logstash indices from before august 1st (T142357) |
[production] |
| 17:04 |
<demon@tin> |
Synchronized wmf-config/: Remove old obsolete ExtensionMessages files (duration: 00m 49s) |
[production] |
| 16:50 |
<godog> |
bounce uwsgi on labmon1001 T143556 |
[production] |
| 16:15 |
<ema@palladium> |
conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe']) |
[production] |
| 16:15 |
<ema@palladium> |
conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx']) |
[production] |
| 15:58 |
<papaul> |
!log wdqs200[1-2] - signing puppet certs, salt-key, initial run |
[production] |
| 15:41 |
<ema@palladium> |
conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe']) |
[production] |
| 15:41 |
<ema@palladium> |
conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx']) |
[production] |
| 15:26 |
<_joe_> |
attempting to restart apache on rhodium, swapping, load exploding |
[production] |
| 15:22 |
<papaul> |
installing wdqs200[1-2] |
[production] |
| 14:41 |
<demon@tin> |
Synchronized multiversion/getMWVersion.php: fixup error message (duration: 00m 47s) |
[production] |
| 13:53 |
<gehel> |
rolling restart of logstash nodes to validate fix to T142357 |
[production] |
| 13:50 |
<hashar> |
European SWAT done |
[production] |
| 13:49 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: Fully restrict uploads on ms.wikipedia T141227 (duration: 00m 46s) |
[production] |
| 13:48 |
<zfilipin@tin> |
Synchronized dblists/commonsuploads.dblist: Fully restrict uploads on ms.wikipedia T141227 (duration: 00m 46s) |
[production] |
| 13:39 |
<ema> |
upgrading cp4005 to Varnish 4 (T131502) |
[production] |
| 13:27 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: Make default ORES threshold soft (higher threshold) T143738 (duration: 00m 59s) |
[production] |
| 12:56 |
<mobrovac> |
changeprop deploying c5fd932 |
[production] |
| 12:29 |
<gehel> |
remove old logstash indices that were not deleted after 31 days |
[production] |
| 11:10 |
<godog> |
enable collection of mysqld metrics from prometheus2002 too |
[production] |
| 10:49 |
<moritzm> |
start installing hhvm updates in eqiad |
[production] |
| 10:13 |
<gehel> |
increase recovery throttling on elasticsearch codfw to reduce rolling restart time |
[production] |
| 09:52 |
<moritzm> |
disabled creation of unprivileged user namespaces on trusty systems via sysctl kernel.unprivileged_userns_clone |
[production] |
| 09:30 |
<gehel> |
starting rolling restart of elasticearch codfw for JVM and elasticsearch upgrade |
[production] |
| 09:03 |
<hashar> |
gallium contint::firewall: Limited to production networks https://gerrit.wikimedia.org/r/301627 . For monitoring do: grep iptables-dropped /var/log/syslog |
[production] |
| 08:39 |
<ema> |
dns: re-enabling codfw |
[production] |
| 08:29 |
<paravoid> |
disabling cr2-codfw:xe-5/0/1 (link to cr2-eqiad), flapping since 07:31 UTC |
[production] |
| 08:09 |
<ema> |
disabling codfw in dns |
[production] |
| 03:07 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Aug 24 03:07:38 UTC 2016 (duration 6m 55s) |
[production] |
| 03:00 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.16) (duration: 18m 24s) |
[production] |
| 02:25 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 16s) |
[production] |
| 01:57 |
<mutante> |
temp stop dhcp service on install2001 - debug |
[production] |
|
2016-08-23
§
|
| 23:45 |
<dereckson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable Related Articles on fr.wikinews (T143480) (duration: 00m 53s) |
[production] |
| 23:28 |
<mutante> |
restarted grrrit-wm, apache and gerrit on lead |
[production] |
| 23:23 |
<mutante> |
gerrit restarting for config changes 303355, 304977 |
[production] |
| 22:04 |
<bd808> |
Updated tin:/srv/deployment/logstash/plugins to d18b1c6 (Add output plugin for Sentry) |
[production] |
| 21:27 |
<SMalyshev> |
redeploying WDQS GUI to fix examples breakage |
[production] |
| 21:27 |
<dzahn@palladium> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=dns,name=hydrogen.wikimedia.org |
[production] |
| 21:05 |
<mutante> |
hydrogen - reinstall finished, re-added to salt, restarted ntpd |
[production] |
| 20:22 |
<mutante> |
hydrogen - reinstalling one more time, wrong partitioning |
[production] |
| 19:55 |
<mutante> |
re-signing new puppet certs for hydrogen, initial run, new salt key |
[production] |
| 19:34 |
<hashar> |
1.28.0-wmf.16 to group0 looks successful. |
[production] |
| 19:19 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: fix compile & sync of wikiversions |
[production] |
| 19:17 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 50s) |
[production] |
| 19:13 |
<hashar@tin> |
Synchronized wikiversions.json: Group0 to 1.28.0-wmf.16 T141551 (duration: 00m 48s) |
[production] |
| 19:04 |
<mutante> |
rebooting hydrogen |
[production] |