|
2019-07-02
ยง
|
| 10:15 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:05 |
<elukey> |
powercycle analytics1056 (soft lockups logged in the serial console, no ssh, no net connectivity) |
[production] |
| 10:05 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:05 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:02 |
<ema> |
pool cp2020 w/ ATS backend T226637 |
[production] |
| 10:00 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:00 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:58 |
<godog> |
restart rsyslog on wezen - T199406 |
[production] |
| 09:55 |
<moritzm> |
draining restbase1025 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 09:53 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:53 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:50 |
<vgutierrez> |
rebooting secondary lvs servers for MDS security updates |
[production] |
| 09:48 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:46 |
<moritzm> |
draining restbase1024 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 09:39 |
<marostegui> |
Upgrade db2094 (codfw sanitarium) T227062 |
[production] |
| 09:39 |
<moritzm> |
draining restbase1023 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 09:34 |
<marostegui> |
Upgrade mysql on 2080 db2081 db2083 - T227062 |
[production] |
| 09:29 |
<moritzm> |
draining restbase1022 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 09:28 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:22 |
<moritzm> |
draining restbase1021 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 09:11 |
<moritzm> |
draining restbase1020 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 09:00 |
<moritzm> |
draining restbase1019 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 08:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:56 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:55 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:55 |
<ema> |
depool cp2020 and reimage as upload_ats T226637 |
[production] |
| 08:52 |
<moritzm> |
draining restbase1018 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 08:50 |
<ema> |
pool cp2018 w/ ATS backend T226637 |
[production] |
| 08:36 |
<moritzm> |
draining restbase1017 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 08:20 |
<moritzm> |
draining restbase1016 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 08:19 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:19 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:10 |
<godog> |
restbase spare hosts, mask and stop restbase - T227054 |
[production] |
| 07:58 |
<moritzm> |
draining restbase2020 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 07:55 |
<ema> |
depool cp2018 and reimage as upload_ats T226637 |
[production] |
| 07:48 |
<moritzm> |
draining restbase2019 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 07:48 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:47 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:47 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:20 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1092 (duration: 00m 49s) |
[production] |
| 07:05 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1092 (duration: 00m 49s) |
[production] |
| 05:48 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1092 into API (duration: 00m 49s) |
[production] |
| 05:37 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1092 (duration: 00m 48s) |
[production] |
| 05:23 |
<marostegui> |
Upgrade MySQL and kernel on db1092 |
[production] |
| 05:23 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 54s) |
[production] |
| 01:39 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@b8a496b]: fix private sqoop (duration: 17m 36s) |
[production] |