|
2020-08-31
§
|
| 12:54 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) |
[production] |
| 12:54 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.02-restore-ttl |
[production] |
| 12:53 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) |
[production] |
| 12:53 |
<oblivian@cumin2001> |
Switching services parsoid: eqiad => codfw |
[production] |
| 12:53 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.01-switch-dc |
[production] |
| 12:53 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) |
[production] |
| 12:47 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 12:45 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) |
[production] |
| 12:45 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.02-restore-ttl |
[production] |
| 12:44 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) |
[production] |
| 12:44 |
<oblivian@cumin2001> |
Switching services restbase-async: eqiad => codfw |
[production] |
| 12:44 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.01-switch-dc |
[production] |
| 12:43 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) |
[production] |
| 12:37 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 12:14 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) |
[production] |
| 12:14 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.02-restore-ttl |
[production] |
| 12:13 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) |
[production] |
| 12:13 |
<oblivian@cumin2001> |
Switching services restbase-async: eqiad => codfw |
[production] |
| 12:13 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.01-switch-dc |
[production] |
| 12:10 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) |
[production] |
| 12:05 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 11:58 |
<elukey> |
reimage kafka-jumbo1001 to Buster |
[production] |
| 11:32 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: 5d583d9550787a8e36c29ca841233615405fcb7e: Disable MediaSearch A/B test (duration: 00m 55s) |
[production] |
| 11:22 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 81f88fde2aad23a619047b1177a6188f51df11a9: Enable Signature button on Wikiproject for hywiki (T261550) (duration: 00m 54s) |
[production] |
| 11:22 |
<jbond42> |
removing old hiera version 1 and 3 backends |
[production] |
| 11:20 |
<arturo> |
[codfw1dev] livehacking https://gerrit.wikimedia.org/r/c/operations/puppet/+/615161 in the puppetmasters for tests before merging |
[admin] |
| 11:17 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: b74893fecdaae599077daad5b1219ad3b9bc7fc9: Enable sitenotice on mobile for closed wikis (T261357) (duration: 00m 56s) |
[production] |
| 11:02 |
<volans> |
upgraded spicerack to 0.0.41 on cumin hosts |
[production] |
| 10:27 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 09:51 |
<elukey> |
executed /srv/phab/phabricator/bin/remove destroy @klausman on phab1001 (following https://wikitech.wikimedia.org/wiki/Phabricator#Delete_a_user) to clear incosistent state of new account (wrong email address) |
[production] |
| 08:43 |
<moritzm> |
installing bind9 security updates on stretch/buster (client-side tools/libs only) |
[production] |
| 07:53 |
<volans> |
uploaded spicerack_0.0.41 to apt.wikimedia.org buster-wikimedia |
[production] |
| 07:30 |
<moritzm> |
installing squid security updates |
[production] |
| 07:24 |
<moritzm> |
installing openexr security updates on buster |
[production] |
| 07:13 |
<elukey> |
run kafka preferred-replica-election on Jumbo after jumbo1005's reimage |
[analytics] |
| 07:12 |
<marostegui> |
Sanitize jawikivoyage on db2094:3325 and db1124:3325 T260482 |
[production] |
| 06:30 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:27 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:06 |
<elukey> |
reimage kafka-jumbo1005 to Debian Buster |
[production] |
| 05:21 |
<marostegui> |
Reload haproxy on dbproxy1017 and dbproxy1021 to test db1128 |
[production] |
|
2020-08-30
§
|
| 21:10 |
<wm-bot> |
<lokal-profil> Manually triggered update_monuments (again) |
[tools.heritage] |
| 21:06 |
<wm-bot> |
<lokal-profil> Forced re-install of pywikibot over pip |
[tools.heritage] |
| 20:40 |
<wm-bot> |
<lokal-profil> Bumped check_emailable_users to 2020 and re-enabled cron job |
[tools.heritage] |
| 20:36 |
<wm-bot> |
<lokal-profil> Manually triggered update_monuments |
[tools.heritage] |
| 20:19 |
<wm-bot> |
<lokal-profil> rename existing .venv and re-run build.sh |
[tools.heritage] |
| 16:13 |
<herron> |
restarted eqiad v5 logstashes |
[production] |
| 11:32 |
<wm-bot> |
<lokal-profil> Deploy latest from Git master: 0955c33 (T224405) |
[tools.heritage] |
| 00:57 |
<Krenair> |
also ran qconf -ds on each |
[tools] |
| 00:34 |
<Krenair> |
Tidied up SGE problems (it was spamming root@ every minute for hours) following host deletions some hours ago - removed tools-sgeexec-0921 through 0931 from @general, ran qmod -rj on all jobs registered for those nodes, then qdel -f on the remainders, then qconf -de on each deleted node |
[tools] |