|
2018-05-02
§
|
| 08:54 |
<moritzm> |
reimaging mw1250, mw1254, mw1255 (app servers) to stretch |
[production] |
| 08:36 |
<moritzm> |
reimaging mw1228, mw1229, mw1230 to stretch (those were logged to SAL before, but failed with IPMI issues before) |
[production] |
| 08:22 |
<ema> |
varnish 5.1.3-1wm8 uploaded to apt.w.o T192368 |
[production] |
| 08:11 |
<elukey> |
upgrading Druid to 0.10 on druid100[4-6] (wikistats 2 backend) - T164008 |
[production] |
| 07:42 |
<elukey> |
remove openjdk-7 related packages from druid100[1-3] after zookeeper upgrade |
[production] |
| 07:36 |
<gehel> |
elasticsearch eqiad rolling restart for plugin update and NUMA config - T191543 / T191236 |
[production] |
| 07:31 |
<elukey> |
upgrade zookeeper on druid100[1-3] to 3.4.9 - T164008 |
[production] |
| 07:27 |
<jynus> |
restart db1098 for upgrade and validation |
[production] |
| 02:44 |
<l10nupdate@tin> |
scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 11s) |
[production] |
| 00:17 |
<ebernhardson> |
start reindex for commonswiki, eqiad elasticsearch, commonswiki_general appears to have failed previous reindex |
[production] |
|
2018-05-01
§
|
| 22:29 |
<ejegg> |
updated CiviCRM from 46883844a3 to 401344fb30 |
[production] |
| 22:19 |
<andrewbogott> |
rebooting labnet1002 for T193579 |
[production] |
| 22:04 |
<krinkle@tin> |
Synchronized php-1.32.0-wmf.1/extensions/EducationProgram/resources/: I7ca59823ffbf2 (duration: 01m 16s) |
[production] |
| 22:01 |
<awight@tin> |
Finished deploy [ores/deploy@bf182e2]: Rollback ores1001 to master (duration: 01m 13s) |
[production] |
| 22:00 |
<awight@tin> |
Started deploy [ores/deploy@bf182e2]: Rollback ores1001 to master |
[production] |
| 21:57 |
<krinkle@tin> |
Synchronized php-1.32.0-wmf.1/includes/libs/rdbms/: Iba663c58224af9 (duration: 01m 19s) |
[production] |
| 21:54 |
<awight@tin> |
Finished deploy [ores/deploy@52347e0]: Test LFS deployment for ORES; T180627 (duration: 03m 21s) |
[production] |
| 21:50 |
<awight@tin> |
Started deploy [ores/deploy@52347e0]: Test LFS deployment for ORES; T180627 |
[production] |
| 21:48 |
<awight@tin> |
Finished deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627 (duration: 00m 26s) |
[production] |
| 21:48 |
<awight@tin> |
Started deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627 |
[production] |
| 21:48 |
<awight@tin> |
Started deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627 |
[production] |
| 21:44 |
<demon@tin> |
Finished scap: group0 to wmf.2 (duration: 68m 18s) |
[production] |
| 21:42 |
<XioNoX> |
re-enabling eqsin-codfw link |
[production] |
| 21:23 |
<urandom> |
restbase: begin culling leaked revisions, enwiki_T_mobile__ng_{lead,remaining} - T192689 |
[production] |
| 20:43 |
<urandom> |
restbase: begin culling leaked revisions, commons_T_mobile__ng_remaining - T192689 |
[production] |
| 20:35 |
<demon@tin> |
Started scap: group0 to wmf.2 |
[production] |
| 20:35 |
<ebernhar|lunch> |
update stale apifeatureusage-search-svc-eqiad-wmnet template in eqiad elasticsearch and delete unused apifeatureusage template |
[production] |
| 20:28 |
<urandom> |
restbase: begin culling leaked revisions, commons_T_mobile__ng_lead - T192689 |
[production] |
| 20:06 |
<demon@tin> |
Pruned MediaWiki: 1.31.0-wmf.27 (duration: 03m 08s) |
[production] |
| 20:00 |
<ottomata> |
rolling restart of eventbus to apply new logstash formatter version T193230 |
[production] |
| 19:59 |
<demon@tin> |
Pruned MediaWiki: 1.31.0-wmf.30 [keeping static files] (duration: 01m 43s) |
[production] |
| 19:49 |
<urandom> |
restbase: begin culling leaked revisions, others_T_mobile__ng_remaining - T192689 |
[production] |
| 19:27 |
<ottomata> |
rolling restart of eventbus to apply logstash tag https://phabricator.wikimedia.org/T193230 |
[production] |
| 19:17 |
<mutante> |
mw2174,mw2175,mw2176 ff - reinstalling with wmf-auto-reimage to stretch |
[production] |
| 19:00 |
<ottomata> |
rolling restart of eventlogging-service-eventbus to apply logstash logging configs - T193230 |
[production] |
| 18:53 |
<ariel@tin> |
Finished deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone (duration: 00m 04s) |
[production] |
| 18:53 |
<ariel@tin> |
Started deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone |
[production] |
| 17:40 |
<XioNoX> |
disabling eqsin<->codfw link for high packet loss on link |
[production] |
| 17:39 |
<otto@tin> |
Finished deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230 (duration: 02m 29s) |
[production] |
| 17:36 |
<otto@tin> |
Started deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230 |
[production] |
| 16:15 |
<ebernhardson> |
T192972 change eqiad elasticsearch disk watermarks from 85/85 to 80/80 to match disk space alerts |
[production] |
| 16:15 |
<herron> |
manually kicked off mirror@sodium:~$ /usr/local/sbin/update-ubuntu-mirror to clear ubuntu mirror out of sync alert |
[production] |
| 15:50 |
<urandom> |
restbase: begin culling leaked revisions, others_T_mobile__ng_lead -- T192689 |
[production] |
| 15:37 |
<ejegg> |
updated SmashPig standalone from a4de12d415 to 99585c8084 |
[production] |
| 13:35 |
<herron> |
restarted hhvm on mw1233 |
[production] |
| 02:47 |
<l10nupdate@tin> |
scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 04s) |
[production] |
| 01:20 |
<ejegg> |
re-enabled fundraising jobs |
[production] |
| 01:01 |
<ejegg> |
updated CiviCRM from 47197006d5 to 46883844a3 |
[production] |
| 00:51 |
<eileen> |
email sent advising of minor interuption |
[production] |
| 00:50 |
<ejegg> |
disabled fundraising jobs for db update |
[production] |