|
2020-08-31
§
|
| 10:27 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 09:51 |
<elukey> |
executed /srv/phab/phabricator/bin/remove destroy @klausman on phab1001 (following https://wikitech.wikimedia.org/wiki/Phabricator#Delete_a_user) to clear incosistent state of new account (wrong email address) |
[production] |
| 08:43 |
<moritzm> |
installing bind9 security updates on stretch/buster (client-side tools/libs only) |
[production] |
| 07:53 |
<volans> |
uploaded spicerack_0.0.41 to apt.wikimedia.org buster-wikimedia |
[production] |
| 07:30 |
<moritzm> |
installing squid security updates |
[production] |
| 07:24 |
<moritzm> |
installing openexr security updates on buster |
[production] |
| 07:13 |
<elukey> |
run kafka preferred-replica-election on Jumbo after jumbo1005's reimage |
[analytics] |
| 07:12 |
<marostegui> |
Sanitize jawikivoyage on db2094:3325 and db1124:3325 T260482 |
[production] |
| 06:30 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:27 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:06 |
<elukey> |
reimage kafka-jumbo1005 to Debian Buster |
[production] |
| 05:21 |
<marostegui> |
Reload haproxy on dbproxy1017 and dbproxy1021 to test db1128 |
[production] |
|
2020-08-30
§
|
| 21:10 |
<wm-bot> |
<lokal-profil> Manually triggered update_monuments (again) |
[tools.heritage] |
| 21:06 |
<wm-bot> |
<lokal-profil> Forced re-install of pywikibot over pip |
[tools.heritage] |
| 20:40 |
<wm-bot> |
<lokal-profil> Bumped check_emailable_users to 2020 and re-enabled cron job |
[tools.heritage] |
| 20:36 |
<wm-bot> |
<lokal-profil> Manually triggered update_monuments |
[tools.heritage] |
| 20:19 |
<wm-bot> |
<lokal-profil> rename existing .venv and re-run build.sh |
[tools.heritage] |
| 16:13 |
<herron> |
restarted eqiad v5 logstashes |
[production] |
| 11:32 |
<wm-bot> |
<lokal-profil> Deploy latest from Git master: 0955c33 (T224405) |
[tools.heritage] |
| 00:57 |
<Krenair> |
also ran qconf -ds on each |
[tools] |
| 00:34 |
<Krenair> |
Tidied up SGE problems (it was spamming root@ every minute for hours) following host deletions some hours ago - removed tools-sgeexec-0921 through 0931 from @general, ran qmod -rj on all jobs registered for those nodes, then qdel -f on the remainders, then qconf -de on each deleted node |
[tools] |
|
2020-08-29
§
|
| 18:05 |
<Amir1> |
end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T261451) |
[production] |
| 17:45 |
<Amir1> |
start of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T261451) |
[production] |
| 16:02 |
<bstorm> |
deleting "tools-sgeexec-0931", "tools-sgeexec-0930", "tools-sgeexec-0929", "tools-sgeexec-0928", "tools-sgeexec-0927" |
[tools] |
| 16:00 |
<bstorm> |
deleting "tools-sgeexec-0926", "tools-sgeexec-0925", "tools-sgeexec-0924", "tools-sgeexec-0923", "tools-sgeexec-0922", "tools-sgeexec-0921" |
[tools] |
| 12:01 |
<James_F> |
dockerfiles: [mediawiki-phan-php73] Publishing 0.1.1 |
[releng] |
| 11:43 |
<James_F> |
layout: Migrate REL1_35 branches to use PHP 7.3 by default |
[releng] |
| 11:24 |
<James_F> |
layout: Migrate use of selenium-only jobs to explicit versions |
[releng] |
|
2020-08-28
§
|
| 22:35 |
<dpifke> |
Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/623068 in beta puppet; should only affect deployment-webperf12. |
[releng] |
| 21:53 |
<ryankemper> |
`sudo systemctl reload nginx.service` on `cloudelastic100[5,6].wikimedia.org` to try to resolve certificate warning issues |
[production] |
| 20:12 |
<bd808> |
Running `wmcs-novastats-dnsleaks --delete` from cloudcontrol1003 |
[admin] |
| 19:17 |
<Amir1> |
restarting codesearch to pick up the new config (T261517) |
[codesearch] |
| 19:12 |
<bstorm> |
moving aside weird old mitaka-jessie sources.list file on cloudinfra-db02 |
[cloudinfra] |
| 19:11 |
<andrewbogott> |
rebooting cloudvirt1006. It's a spare, unused system but showing a bus error and icinga alerts; not worth saving if it needs saving |
[production] |
| 17:39 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 17:39 |
<mutante> |
shutting down mw2196 |
[production] |
| 17:37 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 16:40 |
<rzl> |
switchdc live test complete |
[production] |
| 16:36 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) |
[production] |
| 16:35 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-update-tendril |
[production] |
| 16:35 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) |
[production] |
| 16:34 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-start-maintenance |
[production] |
| 16:33 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) |
[production] |
| 16:33 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restore-ttl |
[production] |
| 16:33 |
<rzl@cumin1001> |
END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99) |
[production] |
| 16:33 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restore-ttl |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) |
[production] |
| 16:29 |
<rzl@cumin1001> |
[DRY-RUN] MediaWiki read-only period ends at: 2020-08-28 16:29:24.432463 |
[production] |
| 16:29 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.07-set-readwrite |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) |
[production] |