|
2019-07-12
ยง
|
| 23:35 |
<mutante> |
netmon1003 - shutdown -h now after it's gone from Icinga now |
[production] |
| 23:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 23:31 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 23:28 |
<mutante> |
netmon1003 - stopping apache2 service (decom of servermon.wikimedia.org) |
[production] |
| 19:41 |
<James_F> |
Disabled 2FA for MSchottlender-WMF for device reset. |
[production] |
| 19:17 |
<shdubsh> |
add prometheus-varnishkafka-exporter 0.1 to apt repo T196066 |
[production] |
| 19:15 |
<urandom> |
bootstrapping restbase1017-c -- T222960 |
[production] |
| 19:08 |
<jeh> |
rebooting cloudvirt1018.eqiad.wmnet T216040 |
[production] |
| 18:53 |
<mutante> |
cp1072 - enabling notifications for service checks in icinga, they were disabled but all green and no SAL/ticket. looked like forgotten from the past |
[production] |
| 18:49 |
<gehel> |
setting CPU governor to performance for wdqs1010 - T225713 |
[production] |
| 18:16 |
<Krinkle> |
Remove bogus Graphite data at frontend.navtiming2.requet (typo from Nov 2018), graphite1004/2003 |
[production] |
| 18:02 |
<urandom> |
bootstrapping restbase1017-b -- T222960 |
[production] |
| 16:32 |
<urandom> |
bootstrapping restbase1017-a -- T222960 |
[production] |
| 16:25 |
<jijiki> |
Rolling restart swift proxy on ms-fe* |
[production] |
| 15:25 |
<jeh> |
rebooting cloudvirt1018.eqiad.wmnet T216040 |
[production] |
| 14:05 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 12:45 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 12:39 |
<fsero> |
recreating ci staging namespaces T227775 |
[production] |
| 12:39 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 12:38 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 12:36 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'main' . |
[production] |
| 12:33 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' . |
[production] |
| 12:33 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' . |
[production] |
| 12:22 |
<fsero> |
recreating eventgate-* and blubberoid staging namespaces T227775 |
[production] |
| 12:22 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'mathoid' for release 'staging' . |
[production] |
| 12:22 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'mathoid' for release 'staging' . |
[production] |
| 12:18 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 12:18 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 12:18 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 12:15 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 12:11 |
<fsero> |
recreating sessionstore,cxserver and mathoid staging namespaces T227775 |
[production] |
| 12:10 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'citoid' for release 'staging' . |
[production] |
| 12:06 |
<fsero> |
recreating citoid staging namespace T227775 |
[production] |
| 12:05 |
<fsero@> |
helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . |
[production] |
| 12:01 |
<fsero> |
recreating termbox staging namespace T227775 |
[production] |
| 11:09 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Switchover db2045 x1 codfw master to db2069 (duration: 00m 51s) |
[production] |
| 10:24 |
<jynus> |
switchover x1 codfw master from db2045 to db2069 T227862 |
[production] |
| 10:23 |
<jynus> |
switchover x1 codfw master from db2045 to db2069 |
[production] |
| 09:43 |
<moritzm> |
shut down ldap-codfw-replica01/ldap-codfw-replica02 (pending reimage) |
[production] |
| 08:18 |
<jijiki> |
enable puppet on mw1222 |
[production] |
| 06:35 |
<vgutierrez> |
upgrading acme-chief to version 0.19 in acme-chief test instances - T225945 |
[production] |
| 06:28 |
<vgutierrez> |
uploaded acme-chief 0.19 to apt.wikimedia.org (buster) - T225945 |
[production] |
| 05:45 |
<elukey> |
sudo -i /usr/local/sbin/restart-php7.2-fpm on mwdebug* to clear opcache |
[production] |
| 01:01 |
<Krinkle> |
mw1342 generated some ~ 11,500 additional PHP errors over a 4 hour period (18:00-22:30 UTC), ref T224491 |
[production] |
| 00:59 |
<Krinkle> |
mw1342 is generating strange PHP erros (php7 only), ref T224491 |
[production] |
| 00:58 |
<urandom> |
bootstrapping restbase1017-a -- T222960 |
[production] |
| 00:50 |
<mutante> |
restbase1018 - restart ferm service |
[production] |
| 00:15 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: e4bd91f71b (duration: 00m 50s) |
[production] |
| 00:13 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: f309856f0912 (duration: 00m 50s) |
[production] |
| 00:03 |
<eevans@deploy1001> |
Finished deploy [cassandra/metrics-collector@df909a1]: deploy logback to restbase1017 (T222960) (duration: 00m 03s) |
[production] |