|
2020-07-09
ยง
|
| 12:38 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 12:38 |
<moritzm> |
rebooting urldownloader1001/2001 for kernel update (failed over, these are now the inactive ones) |
[production] |
| 12:23 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) |
[production] |
| 12:22 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 12:22 |
<moritzm> |
rebooting dbmonitor1001 / tendril.wikimedia.org for kernek update |
[production] |
| 12:11 |
<XioNoX> |
enable asw2-b-eqiad:ae3 (to cloudsw1-c8) - T251632 |
[production] |
| 11:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 11:54 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 11:52 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 11:50 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 11:50 |
<moritzm> |
rebooting debmonitor1001 for kernel update |
[production] |
| 11:42 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.40/extensions/Translate/tag/SpecialPageTranslation.php: 6541d3ff51f52fe8a1bdbfa86022f8d97d6c7680: DeprecatablePropertyArray: Use MW_VERSION instead of array_key_exists (T257531) (duration: 01m 05s) |
[production] |
| 11:28 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 3a7c1c33e58637437f819edf039008a00dc5be27: Rename namespace on kn.wikipedia.org (T255337) (duration: 01m 04s) |
[production] |
| 11:24 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 0a3c1f94a702b527842ed4f34d8bf41b26235e64: Add *.oireachtas.ie to the wgCopyUploadsDomains whitelist for commonswiki (T256543) (duration: 01m 04s) |
[production] |
| 11:19 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 11:17 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 11:10 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:10 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: e6f442c6900524482806aeb1b5162e65bf7c97ac: Enable Quicksurveys for Desktop Improvements Project (T246977) (duration: 01m 06s) |
[production] |
| 11:01 |
<vgutierrez> |
restart ats-tls on cp1085 |
[production] |
| 10:55 |
<_joe_> |
restarting php7.2-fpm on mw1282, workers failing with sigill |
[production] |
| 10:54 |
<_joe_> |
depool mw1282 |
[production] |
| 10:54 |
<mvolz@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:34 |
<mvolz@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:23 |
<_joe_> |
rolling restart the remaining restbases in eqiad, and all of codfw |
[production] |
| 10:22 |
<mvolz@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' . |
[production] |
| 10:09 |
<_joe_> |
restarting restbase on rb1020-22 |
[production] |
| 09:53 |
<_joe_> |
restarting restbase on restbase1024,1023 |
[production] |
| 09:36 |
<_joe_> |
restarting restbase on rb1026,1027 to switch to proton on k8s |
[production] |
| 09:34 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:31 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:28 |
<_joe_> |
restarting restbase on restbase1025 to pick up the switch to k8s of proton |
[production] |
| 09:27 |
<godog> |
bounce thanos-compact on thanos-fe2001 |
[production] |
| 09:07 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.change-distro (exit_code=0) |
[production] |
| 08:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P11828 and previous config saved to /var/cache/conftool/dbconfig/20200709-085228-marostegui.json |
[production] |
| 08:44 |
<marostegui> |
Stop haproxy on dbproxy1017 before upgrading to buster - T255408 |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1136', diff saved to https://phabricator.wikimedia.org/P11827 and previous config saved to /var/cache/conftool/dbconfig/20200709-082355-marostegui.json |
[production] |
| 08:23 |
<moritzm> |
imported osm2pgsql 0.96.0+ds-1~bpo9+1 to "main" component T256877 |
[production] |
| 08:22 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.change-distro |
[production] |
| 08:20 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) |
[production] |
| 08:13 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.stop-cluster |
[production] |
| 08:11 |
<XioNoX> |
disable igmp snooping on msw1-codfw |
[production] |
| 07:59 |
<marostegui> |
Stop db1117:3322 to clone db1084, this will trigger haproxy alerts - T257540 |
[production] |
| 07:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P11825 and previous config saved to /var/cache/conftool/dbconfig/20200709-075749-marostegui.json |
[production] |
| 07:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:15 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P11824 and previous config saved to /var/cache/conftool/dbconfig/20200709-053905-marostegui.json |
[production] |
| 05:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1084 from dbctl', diff saved to https://phabricator.wikimedia.org/P11823 and previous config saved to /var/cache/conftool/dbconfig/20200709-053206-marostegui.json |
[production] |