|
2019-10-24
§
|
| 10:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Adjust s6 weights for db1093 and db1085', diff saved to https://phabricator.wikimedia.org/P9466 and previous config saved to /var/cache/conftool/dbconfig/20191024-101810-marostegui.json |
[production] |
| 09:59 |
<hashar> |
Converting CI jobs to use the new PostBuildScript plugin config | https://gerrit.wikimedia.org/r/#/c/integration/config/+/544907/ | T188398 |
[production] |
| 09:57 |
<hashar> |
Restarting CI Jenkins |
[production] |
| 09:35 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:33 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:14 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:12 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:12 |
<gilles@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T234853 Re-enable performance perception survey on ruwiki (duration: 01m 04s) |
[production] |
| 08:39 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 08:37 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:36 |
<godog> |
roll restart rsyslog in codfw/eqiad to pick up new kafka partitions |
[production] |
| 08:18 |
<godog> |
roll restart rsyslog in ulsfo/esams/eqsin to pick up new kafka partitions |
[production] |
| 08:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2092 for analyze table', diff saved to https://phabricator.wikimedia.org/P9465 and previous config saved to /var/cache/conftool/dbconfig/20191024-081519-marostegui.json |
[production] |
| 07:57 |
<XioNoX> |
reboot mr1-esams |
[production] |
| 07:42 |
<godog> |
bump rsyslog- topics partitions to 6 and roll-restart logstash frontends |
[production] |
| 07:24 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 07:22 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:22 |
<XioNoX> |
drain Telia link on cr2-esams |
[production] |
| 06:32 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=parsoid-php,name=eqiad |
[production] |
| 05:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1097:3315 after compression', diff saved to https://phabricator.wikimedia.org/P9463 and previous config saved to /var/cache/conftool/dbconfig/20191024-052002-marostegui.json |
[production] |
| 05:18 |
<marostegui> |
Run analyze enwiki.revision on db2092 T223151 |
[production] |
| 04:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1097:3315 after compression', diff saved to https://phabricator.wikimedia.org/P9462 and previous config saved to /var/cache/conftool/dbconfig/20191024-045954-marostegui.json |
[production] |
| 04:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1089 from special slaves group and leave it with its original pooling options T223151', diff saved to https://phabricator.wikimedia.org/P9461 and previous config saved to /var/cache/conftool/dbconfig/20191024-045924-marostegui.json |
[production] |
| 04:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1097:3315 after compression', diff saved to https://phabricator.wikimedia.org/P9460 and previous config saved to /var/cache/conftool/dbconfig/20191024-045544-marostegui.json |
[production] |
| 04:48 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 04:48 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 03:55 |
<shdubsh> |
temporarily turn down accept delay on fermium - T235983 |
[production] |
| 00:03 |
<mutante> |
restarting gerrit to increase heap_size from 20G to 32G (T225166 T222391) |
[production] |
|
2019-10-23
§
|
| 22:55 |
<brennen@deploy1001> |
Synchronized php-1.35.0-wmf.3/extensions/AbuseFilter: SWAT: [[gerrit:545620|Unbreak filter edit form (T236286)]] (duration: 01m 05s) |
[production] |
| 22:20 |
<twentyafterfour@deploy1001> |
Finished deploy [phabricator/deployment@e4e2b22]: deploy to phab1001 (currently a warm spare server) (duration: 00m 21s) |
[production] |
| 22:20 |
<twentyafterfour@deploy1001> |
Started deploy [phabricator/deployment@e4e2b22]: deploy to phab1001 (currently a warm spare server) |
[production] |
| 22:20 |
<twentyafterfour@deploy1001> |
Finished deploy [phabricator/deployment@e4e2b22]: deploy to phab1001 (currently a warm spare server) (duration: 00m 05s) |
[production] |
| 22:19 |
<twentyafterfour@deploy1001> |
Started deploy [phabricator/deployment@e4e2b22]: deploy to phab1001 (currently a warm spare server) |
[production] |
| 22:15 |
<twentyafterfour@deploy1001> |
Finished deploy [phabricator/deployment@e4e2b22]: deploy to phab1001 (currently a warm spare server) (duration: 01m 10s) |
[production] |
| 22:14 |
<twentyafterfour@deploy1001> |
Started deploy [phabricator/deployment@e4e2b22]: deploy to phab1001 (currently a warm spare server) |
[production] |
| 22:00 |
<twentyafterfour@deploy1001> |
Finished deploy [phabricator/deployment@e4e2b22]: deploy to phab1001 (currently a warm spare server) (duration: 00m 21s) |
[production] |
| 22:00 |
<twentyafterfour@deploy1001> |
Started deploy [phabricator/deployment@e4e2b22]: deploy to phab1001 (currently a warm spare server) |
[production] |
| 21:32 |
<mutante> |
webperf1002/2002 - starting bacula-fd service that is failed after initial puppet run turning them into backup::hosts |
[production] |
| 21:14 |
<ejegg> |
updated Fundraising python tools from b3c7453be2 to ffc7bf764b |
[production] |
| 20:37 |
<shdubsh> |
restart nagios-nrpe-server on stat1007 |
[production] |
| 18:56 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@3aaabf6]: Minor: fix two scripts (duration: 07m 53s) |
[production] |
| 18:49 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@3aaabf6]: Minor: fix two scripts |
[production] |
| 18:29 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@1110d59]: deploying refinery up to 1110d59c3983bcff4986bce1baf885f05ee06ba5 (duration: 06m 40s) |
[production] |
| 18:22 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@1110d59]: deploying refinery up to 1110d59c3983bcff4986bce1baf885f05ee06ba5 |
[production] |
| 17:31 |
<akosiaris> |
restart varnish-be on cp1089 as a response to HTTP availability alerts. High mailbox lag |
[production] |
| 17:25 |
<akosiaris> |
restart varnish-be on cp1081 as a response to HTTP availability alerts |
[production] |
| 15:55 |
<_joe_> |
restarting pybal on lvs2006, then 2003 for picking up parsoid-php |
[production] |
| 15:32 |
<marostegui> |
Enable slow query log 1/20 on db1089 (enwiki) T223151 |
[production] |
| 14:40 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:39 |
<ema@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |