|
2021-03-19
§
|
| 11:18 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 10:45 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:45 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:42 |
<moritzm> |
installing dbmonitor1002 T224589 |
[production] |
| 10:42 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:42 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:41 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:41 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:11 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 10:10 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 10:05 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 10:04 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 09:40 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 09:36 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 08:54 |
<Majavah> |
remove deployment-restbase02 from cassandra and shut it down T250574 |
[releng] |
| 08:45 |
<Majavah> |
disable puppet and stop restbase service on deployment-restbase02 for T250574 |
[releng] |
| 08:22 |
<elukey> |
upload alluxio 2.4.1 to thirdparty/bigtop15 on stretch/buster-wikimedia |
[production] |
| 07:16 |
<ryankemper> |
T275885 `ryankemper@cumin1001:~$ sudo cumin 'P{relforge*}' 'sudo run-puppet-agent'` (change hadn't been merged when I ran the agent earlier) |
[production] |
| 04:04 |
<eileen> |
civicrm revision changed from 99bf1c9210 to 39d24e8b0a, config revision is 26b02db7ba |
[production] |
| 03:27 |
<ryankemper> |
[wdqs] `ryankemper@wdqs1013:~$ sudo systemctl restart wdqs-blazegraph` |
[production] |
| 03:26 |
<ryankemper> |
T275885 `ryankemper@cumin1001:~$ sudo cumin 'P{relforge*}' 'sudo run-puppet-agent'` |
[production] |
| 02:43 |
<ryankemper> |
T275885 Revoking current `relforge` TLS cert in advance of generation of new cert: `ryankemper@puppetmaster1001:/srv/private$ sudo puppet cert clean relforge.svc.eqiad.wmnet` |
[production] |
| 00:51 |
<dancy@deploy1002> |
Synchronized php-1.36.0-wmf.35/extensions/LiquidThreads/classes/Thread.php: T277772 (duration: 00m 58s) |
[production] |
| 00:45 |
<mutante> |
testreduce1001 - stop mysql; rsyncing /var/lib/mysql to /srv/data/mysql (T277580) |
[production] |
| 00:30 |
<bstorm> |
downtimed labstore1004 to check some things in debug mode |
[admin] |
|
2021-03-18
§
|
| 23:56 |
<legoktm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Don't define a default icon (T274199) (duration: 00m 57s) |
[production] |
| 23:38 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.35/includes/user/ActorStore.php: Backport: [[gerrit:673115|ActorStore::getActorById - fall back to master. (T277795)]] (duration: 00m 57s) |
[production] |
| 23:35 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.35/includes/user/ActorStore.php: Backport: [[gerrit:673115|ActorStore::getActorById - fall back to master. (T277795)]] (duration: 00m 58s) |
[production] |
| 23:25 |
<dduvall@deploy1002> |
Synchronized .pipeline: config: [[gerrit:673375|Use build environment HTTP proxy for APT sources (T277109)]] (duration: 01m 02s) |
[production] |
| 23:06 |
<brennen> |
train status: 1.36.0-wmf.35 (T274939) stable on all wikis after deploy of hotfix for T277795 |
[production] |
| 22:53 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.35/includes/specials/SpecialContributions.php: Backport: [[gerrit:673115|ActorStore::getActorById - fall back to master. (T277795)]] (duration: 01m 07s) |
[production] |
| 22:30 |
<dduvall@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 22:29 |
<dduvall@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 22:29 |
<marxarelli> |
deploying https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199 to eqiad/codfw (T277109) |
[releng] |
| 22:28 |
<marxarelli> |
staging https://gerrit.wikimedia.org/r/c/blubber/+/673332 and https://gerrit.wikimedia.org/r/c/blubber/+/671199 |
[releng] |
| 22:25 |
<dduvall@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 20:37 |
<dancy@deploy1002> |
Synchronized php-1.36.0-wmf.35/extensions/LiquidThreads/classes/Thread.php: (no justification provided) (duration: 01m 05s) |
[production] |
| 19:30 |
<razzi> |
rename /usr/lib/python2.7/dist-packages/cqlshlib/copyutil.so back |
[analytics] |
| 19:29 |
<razzi> |
temporarily rename /usr/lib/python2.7/dist-packages/cqlshlib/copyutil.so on aqs1004 to fix https://issues.apache.org/jira/browse/CASSANDRA-11574 |
[analytics] |
| 19:24 |
<bstorm> |
set profile::toolforge::infrastructure across the entire project with login_server set on the bastion and exec node-related prefixes |
[tools] |
| 19:04 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.36.0-wmf.35 |
[production] |
| 19:02 |
<ottomata> |
hdfs dfs -chgrp -R analytics-privatedata-users /wmf/camus - T275396 |
[analytics] |
| 18:55 |
<bstorm> |
set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix |
[toolsbeta] |
| 18:50 |
<arturo> |
deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project) |
[toolsbeta] |
| 18:49 |
<arturo> |
deleting VM toolsbeta-workflow-test, no longer useful |
[toolsbeta] |
| 18:44 |
<arturo> |
replacing toolsbeta-sgegrid-master with a Debian Buster VM (T277653) |
[toolsbeta] |
| 18:28 |
<legoktm> |
re-enabled puppet on registry* |
[production] |
| 18:27 |
<brennen> |
updating gitlab-test to 13.9.4-ce |
[releng] |
| 18:17 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 44eddcc: hrwiki: Deploy Growth features to newcomers (T275684) (duration: 01m 08s) |
[production] |
| 18:12 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 179d9e5: mswiki: Enable Growth features in stealth mode (T277562; 2/2) (duration: 01m 08s) |
[production] |