|
2019-07-02
ยง
|
| 14:02 |
<bblack> |
deploying anycast_healthchecker changes to the recdnses (puppet disabled on all, testing dns4002 first) - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/397723/ |
[production] |
| 13:31 |
<marostegui> |
Upgrade db2086 |
[production] |
| 13:27 |
<ema> |
depool cp2024 and reimage as upload_ats T226637 |
[production] |
| 13:26 |
<XioNoX> |
test fix policy ASXXX_in (missing `then next policy`) |
[production] |
| 13:23 |
<marostegui> |
Upgrade db2085 |
[production] |
| 13:13 |
<XioNoX> |
push RPKI classification to eqiad - T220669 |
[production] |
| 13:09 |
<XioNoX> |
push RPKI classification to eqsin - T220669 |
[production] |
| 13:06 |
<ema> |
pool cp2022 w/ ATS backend T226637 |
[production] |
| 12:51 |
<XioNoX> |
push RPKI classification to AMS - T220669 |
[production] |
| 12:47 |
<marostegui> |
Upgrade db2082 - T227062 |
[production] |
| 12:30 |
<jijiki> |
Power cycle ms-be1021 - T227076 |
[production] |
| 11:51 |
<ema> |
depool cp2022 and reimage as upload_ats T226637 |
[production] |
| 11:40 |
<Urbanecm> |
EU SWAT really done |
[production] |
| 11:37 |
<Urbanecm> |
Ran mwscript resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 86.49.134.37 for T225555 |
[production] |
| 11:37 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[:gerrit:520214|Add new throttle rule for cswiki workshop]] (T225555) (duration: 00m 49s) |
[production] |
| 11:33 |
<Urbanecm> |
Reopen EU SWAT for last-time throttle rule |
[production] |
| 11:33 |
<moritzm> |
re-enabled meta monitoring for icinga2001 |
[production] |
| 11:26 |
<moritzm> |
rebooting icinga2001 for kernel security update |
[production] |
| 11:26 |
<jijiki> |
Run restart-php-fpm in all-mw-eqiad - T223391 |
[production] |
| 11:25 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 11:25 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:23 |
<moritzm> |
temporarily disabled meta monitoring for icinga2001 |
[production] |
| 11:16 |
<dcausse> |
EU Swat done |
[production] |
| 11:15 |
<dcausse@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/CirrusSearch/includes/Updater.php: T226592: Ignore broken redirects when updating incoming link counts (duration: 00m 49s) |
[production] |
| 11:06 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [cirrus] Enable UTR30 as a lookup method for ns prefixes on group1 (duration: 00m 50s) |
[production] |
| 10:47 |
<jijiki> |
Rollout Wikidiff 1.8.2 to eqiad - T223391 |
[production] |
| 10:45 |
<jijiki> |
Rollout Wikidiff 1.8.2 to codfw - T223391 |
[production] |
| 10:32 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:32 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:21 |
<moritzm> |
draining restbase1027 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 10:15 |
<moritzm> |
draining restbase1026 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 10:15 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:15 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:05 |
<elukey> |
powercycle analytics1056 (soft lockups logged in the serial console, no ssh, no net connectivity) |
[production] |
| 10:05 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:05 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:02 |
<ema> |
pool cp2020 w/ ATS backend T226637 |
[production] |
| 10:00 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:00 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:58 |
<godog> |
restart rsyslog on wezen - T199406 |
[production] |
| 09:55 |
<moritzm> |
draining restbase1025 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 09:53 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:53 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:50 |
<vgutierrez> |
rebooting secondary lvs servers for MDS security updates |
[production] |
| 09:48 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:46 |
<moritzm> |
draining restbase1024 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 09:39 |
<marostegui> |
Upgrade db2094 (codfw sanitarium) T227062 |
[production] |
| 09:39 |
<moritzm> |
draining restbase1023 for eventual reboot for MDS security updates / OpenJDK security update |
[production] |
| 09:34 |
<marostegui> |
Upgrade mysql on 2080 db2081 db2083 - T227062 |
[production] |