|
2019-07-17
ยง
|
| 15:11 |
<papaul> |
shutting down mw2250 for disk replacement |
[production] |
| 15:10 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 15:07 |
<hashar> |
upgrading CI Jenkins # T228142 |
[production] |
| 15:06 |
<papaul> |
shutting down ms-be2022 for HW troubleshooting |
[production] |
| 15:06 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:05 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:03 |
<jijiki> |
Depool mw2269 to reboot it - T227548 |
[production] |
| 15:00 |
<godog> |
poweroff ms-be2022 - T227667 |
[production] |
| 14:55 |
<moritzm> |
updated jenkins in thirdparty/ci (stretch) and thirdparty (jessie) to 2.176.2 (T228142) |
[production] |
| 14:45 |
<fsero> |
enabling container-sync logging T228196 |
[production] |
| 14:41 |
<otto@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:41 |
<otto@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:35 |
<moritzm> |
restart pybal on lvs2002 (codfw primary) T227778 |
[production] |
| 14:32 |
<gehel@cumin1001> |
START - Cookbook sre.postgresql.postgres-init |
[production] |
| 14:30 |
<gehel> |
repool maps1004 - T218097 |
[production] |
| 14:11 |
<liw@deploy1001> |
Synchronized php: group1 wikis to 1.34.0-wmf.14 (duration: 00m 54s) |
[production] |
| 14:10 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.14 |
[production] |
| 14:09 |
<moritzm> |
restarting pybal on backup LVSes in codfw |
[production] |
| 14:02 |
<liw@deploy1001> |
Synchronized php-1.34.0-wmf.14/extensions/CirrusSearch/includes/Searcher.php: Do not serialize ResultsType instance T228276 (duration: 00m 55s) |
[production] |
| 13:37 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 13:26 |
<moritzm> |
disabled puppet on Icinga hosts in preparation of adding the LDAP replicas/codfw to LVS |
[production] |
| 13:10 |
<ema> |
cp-codfw: varnish frontend rolling restarts for 5.1.3-1wm11 upgrades T227672 |
[production] |
| 13:06 |
<ema> |
prometheus servers: remove varnish-upload_$dc_backend.yaml, replaced by ATS equivalent T227668 |
[production] |
| 12:57 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 12:36 |
<godog> |
upgrade hp raid firmware on ms-be1 hosts - T141756 |
[production] |
| 12:15 |
<Urbanecm> |
Running foreachwiki extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php in tmux session on mwmaint1002 (T209565) |
[production] |
| 12:11 |
<Urbanecm> |
Ran extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php for cawiki and viwiki (T209565) |
[production] |
| 11:58 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 11:30 |
<mlitn@deploy1001> |
Synchronized php-1.34.0-wmf.14/extensions/WikibaseMediaInfo: [WikibaseMediaInfo] Revert "Add Wikidata links to statement UI elements" (duration: 00m 56s) |
[production] |
| 11:16 |
<dcausse> |
reindexing wikidata (elastic@eqiad) T227136 |
[production] |
| 11:08 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T227136: [cirrus] switch search traffic (except completion) to codfw (duration: 00m 54s) |
[production] |
| 10:53 |
<moritzm> |
re-enabled icinga1001 in meta monitoring |
[production] |
| 10:41 |
<godog> |
install updated linux-image-4.9.0-9-amd64 on ms-be hosts |
[production] |
| 10:30 |
<godog> |
start rolling reboot of ms-be eqiad hosts - T225713 |
[production] |
| 10:30 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 10:23 |
<moritzm> |
rebooting icinga1001 for kernel update |
[production] |
| 10:20 |
<moritzm> |
disabled icinga1001 in meta monitoring |
[production] |
| 10:18 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 10:18 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:08 |
<moritzm> |
rebooting lithium for kernel update |
[production] |
| 10:04 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:04 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:33 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 09:33 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) |
[production] |
| 09:23 |
<moritzm> |
rebooting grafana1001 to pick up MDS-enabled qemu |
[production] |
| 09:21 |
<ema> |
cp-ats: upgrade fifo-log-demux to 0.3 T227668 |
[production] |
| 09:21 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool and clarify db2045 status T227862 (duration: 00m 55s) |
[production] |
| 09:19 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:19 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:15 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |