|
2017-09-20
ยง
|
| 16:04 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch |
[production] |
| 16:03 |
<gehel> |
depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw) |
[production] |
| 15:46 |
<ejegg> |
limited donations queue consumer to 100 per run |
[production] |
| 15:36 |
<ejegg> |
updated payments-wiki from 896f16ac380aaa6022749b395f9187ecf4bc802b to 9da84823ff4ac3071d218ec211fb59c054de9e15 |
[production] |
| 15:36 |
<gehel> |
upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231 |
[production] |
| 15:33 |
<ejegg> |
limited donations queue consumer to 200 per run |
[production] |
| 14:03 |
<akosiaris@tin> |
Synchronized wmf-config/InitialiseSettings.php: Whitelist wtp10[25-48] for Linter (duration: 00m 49s) |
[production] |
| 13:42 |
<otto@tin> |
Synchronized /srv/mediawiki-staging/wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 49s) |
[production] |
| 13:24 |
<ema> |
begin rolling codfw varnish-be restarts, 20m spacing |
[production] |
| 13:20 |
<dereckson@tin> |
Synchronized wmf-config/CommonSettings.php: Switch elasticsearch active cluster to codfw ([[Gerrit:378850]], 3/3) (duration: 00m 48s) |
[production] |
| 13:18 |
<dereckson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Switch elasticsearch active cluster to codfw ([[Gerrit:378850]], 2/3) (duration: 00m 49s) |
[production] |
| 13:17 |
<dereckson@tin> |
Synchronized tests/cirrusTest.php: Switch elasticsearch active cluster to codfw ([[Gerrit:378850]], 1/3, no-op in prod) (duration: 00m 49s) |
[production] |
| 12:51 |
<ema> |
restarted varnish-be on cp4028 |
[production] |
| 12:28 |
<gehel> |
upgrading nodejs to 6.11 on maps-test2003 for testing - T171707 |
[production] |
| 12:19 |
<ema> |
restarted varnish-be on cp4027 |
[production] |
| 12:18 |
<moritzm> |
uploaded apache 2.4.10-10+deb8u11+wmf1 to jessie-wikimedia (rebuild of latest security update with our local patches on top) |
[production] |
| 12:08 |
<hoo@tin> |
Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717) (duration: 00m 49s) |
[production] |
| 11:58 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 (duration: 07m 20s) |
[production] |
| 11:56 |
<ema> |
restarted varnish-be on cp4018 |
[production] |
| 11:51 |
<mobrovac@tin> |
Started deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 |
[production] |
| 11:49 |
<jynus> |
stopping replication on db1101 for faster repartition work T176311 |
[production] |
| 11:45 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: depool db1101 (duration: 00m 58s) |
[production] |
| 11:34 |
<ema> |
restarted varnish-be on cp4010 |
[production] |
| 11:25 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@dea2b41]: (no justification provided) (duration: 00m 02s) |
[production] |
| 11:25 |
<mobrovac@tin> |
Started deploy [restbase/deploy@dea2b41]: (no justification provided) |
[production] |
| 11:03 |
<moritzm> |
rolling out new debdeploy release across the fleet |
[production] |
| 11:02 |
<mobrovac@tin> |
Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 19s) |
[production] |
| 11:02 |
<mobrovac@tin> |
Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) |
[production] |
| 11:02 |
<ema> |
restarted varnish-be on cp4008 |
[production] |
| 11:02 |
<mobrovac@tin> |
Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s) |
[production] |
| 11:01 |
<mobrovac@tin> |
Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) |
[production] |
| 11:01 |
<mobrovac@tin> |
Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 03s) |
[production] |
| 11:01 |
<mobrovac@tin> |
Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) |
[production] |
| 11:00 |
<mobrovac@tin> |
Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s) |
[production] |
| 11:00 |
<mobrovac@tin> |
Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) |
[production] |
| 09:55 |
<bblack> |
restarted varnish-be on cp3042 |
[production] |
| 09:29 |
<bblack> |
restarted varnish-be on cp1052 |
[production] |
| 09:23 |
<ema> |
restarted varnish-be on cp3043 |
[production] |
| 09:21 |
<ema> |
restarted varnish-be on cp3041 |
[production] |
| 09:19 |
<ema> |
restarted varnish-be on cp3032 |
[production] |
| 09:14 |
<ema> |
restarted varnish-be on cp3040 |
[production] |
| 08:42 |
<bblack> |
restarted varnish-be on cp1053 |
[production] |
| 08:41 |
<bblack> |
restarted varnish-be on cp1068 |
[production] |
| 08:40 |
<bblack> |
restarted varnish-be on cp1054 |
[production] |
| 08:40 |
<bblack> |
restarted varnish-be on cp1065 |
[production] |
| 07:11 |
<ema> |
cp3030 varnish-be restart due to 503s |
[production] |
| 03:16 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Sep 20 03:16:25 UTC 2017 (duration 7m 31s) |
[production] |
| 03:08 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 57s) |
[production] |
| 02:31 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 06s) |
[production] |
| 00:55 |
<bblack> |
cp1066 - backend restart, mailbox lag, text |
[production] |