|
2017-09-20
ยง
|
| 23:49 |
<catrope@tin> |
Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s) |
[production] |
| 23:47 |
<catrope@tin> |
Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s) |
[production] |
| 23:40 |
<mutante> |
phabricator: aphlict (notification service) now running on prod server |
[production] |
| 23:36 |
<mutante> |
phab1001 - re-enable puppet, run after follow-up fix, adding notification server config |
[production] |
| 23:31 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287) (duration: 00m 48s) |
[production] |
| 23:29 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694) (duration: 00m 48s) |
[production] |
| 23:21 |
<catrope@tin> |
Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s) |
[production] |
| 23:14 |
<mutante> |
phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group |
[production] |
| 23:13 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079) (duration: 00m 48s) |
[production] |
| 23:11 |
<mutante> |
phab2001 - groupadd aphlict,temp adduser aphlict to debug user creation issue |
[production] |
| 22:22 |
<ebernhardson> |
elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb |
[production] |
| 21:54 |
<mutante> |
phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001 |
[production] |
| 21:45 |
<eileen> |
update civicrm from 0fb46767f0c86b3fc85c44e228fd7c1a7ae75e46 to da833ac561f7586a0b64fbef2cdf2963bc4055c3 |
[production] |
| 21:09 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet |
[production] |
| 20:58 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet |
[production] |
| 20:40 |
<bsitzmann@tin> |
Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) (duration: 06m 05s) |
[production] |
| 20:36 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet |
[production] |
| 20:35 |
<ebernhar|lunch> |
increase max recovery bytes/s to 80mb in eqiad elasitcsearch cluster |
[production] |
| 20:34 |
<bsitzmann@tin> |
Started deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) |
[production] |
| 20:33 |
<demon@tin> |
Synchronized php-1.30.0-wmf.19/includes/diff/DifferenceEngine.php: fix some warnings (duration: 00m 51s) |
[production] |
| 19:55 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=cp4016.ulsfo.wmnet |
[production] |
| 19:54 |
<eevans@tin> |
Finished deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster (duration: 00m 10s) |
[production] |
| 19:54 |
<eevans@tin> |
Started deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster |
[production] |
| 19:53 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet |
[production] |
| 19:48 |
<eevans@tin> |
Finished deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release (duration: 00m 47s) |
[production] |
| 19:47 |
<eevans@tin> |
Started deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release |
[production] |
| 19:39 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=cp4008.ulsfo.wmnet |
[production] |
| 19:26 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.19 |
[production] |
| 19:24 |
<demon@tin> |
Synchronized php: symlink swap for wmf.19 (duration: 00m 49s) |
[production] |
| 19:02 |
<catrope@tin> |
Finished scap: Patches for T176302, T175962, T173533 (duration: 21m 29s) |
[production] |
| 18:40 |
<catrope@tin> |
Started scap: Patches for T176302, T175962, T173533 |
[production] |
| 17:29 |
<hoo> |
Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on terbium. Can be KILLed at any time, if needed. |
[production] |
| 17:29 |
<hoo> |
Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on elwiki. Can be KILLed at any time, if needed. |
[production] |
| 17:13 |
<hoo> |
Ran mwscript refreshLinks.php --wiki elwiki --namespace 0 -e 30 for testing purposes |
[production] |
| 17:06 |
<ejegg> |
removed limit from donation queue consumer |
[production] |
| 17:05 |
<otto@tin> |
Finished deploy [eventstreams/deploy@e62ab64]: (no justification provided) (duration: 04m 44s) |
[production] |
| 17:01 |
<otto@tin> |
Started deploy [eventstreams/deploy@e62ab64]: (no justification provided) |
[production] |
| 16:53 |
<madhuvishy> |
Upgraded and restarted apache2 on labs-puppetmaster.wikimedia.org |
[production] |
| 16:45 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch |
[production] |
| 16:42 |
<gehel> |
rolling restart of logstash ingesters after cirrus eqiad recovery |
[production] |
| 16:34 |
<gehel> |
elasticsearch eqiad is finally recovering, after the restart of elastic1051 |
[production] |
| 16:34 |
<gehel> |
cleanup jar hell and restart elastic1051 |
[production] |
| 16:19 |
<gehel> |
starting elasticsearch on all nodes on eqiad |
[production] |
| 16:10 |
<gehel> |
restarting elasticsearch masters on eqiad (elastic1030, 36 and 40) |
[production] |
| 16:04 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch |
[production] |
| 16:03 |
<gehel> |
depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw) |
[production] |
| 15:46 |
<ejegg> |
limited donations queue consumer to 100 per run |
[production] |
| 15:36 |
<ejegg> |
updated payments-wiki from 896f16ac380aaa6022749b395f9187ecf4bc802b to 9da84823ff4ac3071d218ec211fb59c054de9e15 |
[production] |
| 15:36 |
<gehel> |
upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231 |
[production] |
| 15:33 |
<ejegg> |
limited donations queue consumer to 200 per run |
[production] |