|
2021-09-23
ยง
|
| 16:38 |
<ryankemper> |
T280001 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/713959, running puppet on `*w*qs*` (i.e. wcqs and wdqs) |
[production] |
| 16:13 |
<elukey> |
reboot an-worker1096 to see if megacli status for a new disk changes - T290805 |
[production] |
| 16:09 |
<brennen> |
gitlab1001: reverting [[gerrit:714382|gitlab cas: uid instead of CN; add nickname_key]] for T288392, as existing user logins are broken. |
[production] |
| 15:54 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@mwmaint1002:~$ echo 'https://query.wikidata.org/querybuilder/' | mwscript purgeList.php # T285761 |
[production] |
| 15:54 |
<brennen> |
gitlab1001: brief downtime to apply [[gerrit:714382|gitlab cas: uid instead of CN; add nickname_key]] for T288392 |
[production] |
| 15:12 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:09 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:09 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 15:06 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:06 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 14:58 |
<reedy@deploy1002> |
Synchronized wmf-config/reverse-proxy-staging.php: T291643 (duration: 01m 05s) |
[production] |
| 14:19 |
<moritzm> |
removed routers filter for mx1001, reimage to bullseye complete T286911 |
[production] |
| 14:19 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:19 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 14:14 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 13:53 |
<effie> |
upgrade php7.2 on codfw - T291052 |
[production] |
| 13:36 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for mx1001.wikimedia.org: Renew puppet certificate - jmm@cumin2002 |
[production] |
| 13:36 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.renew-cert for mx1001.wikimedia.org: Renew puppet certificate - jmm@cumin2002 |
[production] |
| 13:34 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for mx1001.wikimedia.org: Renew puppet certificate - jmm@cumin2002 |
[production] |
| 13:34 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.renew-cert for mx1001.wikimedia.org: Renew puppet certificate - jmm@cumin2002 |
[production] |
| 13:28 |
<marostegui> |
Deploy schema change on s8 codfw wikidatawiki.wb_changes T291584 |
[production] |
| 13:27 |
<moritzm> |
reimaging mx1001 to bullseye T286911 |
[production] |
| 13:25 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx1001.wikimedia.org with reason: reimage |
[production] |
| 13:25 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mx1001.wikimedia.org with reason: reimage |
[production] |
| 13:23 |
<jbond> |
merge refactor of resolv.conf puppet class - (gerrit 717241) |
[production] |
| 13:14 |
<marostegui> |
Deploy schema change on s4 {commonswiki,testcommonswiki}.wb_changes T291584 |
[production] |
| 13:11 |
<marostegui> |
Deploy schema change on s3 testwikidatawiki.wb_changes T291584 |
[production] |
| 13:09 |
<elukey> |
update pcc facts (after change in puppetdb's fact filter list, to allow partitions for analytics) |
[production] |
| 11:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:19 |
<marostegui> |
Upgrade db2081 db2082 db2083 db2084 db2091 db2152 T290868 |
[production] |
| 11:16 |
<kostajh> |
UTC morning backport and config deploys done |
[production] |
| 11:15 |
<kharlan@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:722961|GrowthExperiments: Place new dewiki accounts in control group (T288420)]] (duration: 01m 06s) |
[production] |
| 11:10 |
<jynus> |
restart and upgrade db2141 T290865 |
[production] |
| 10:55 |
<mvolz@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'zotero' for release 'production' . |
[production] |
| 10:53 |
<moritzm> |
mx1001 filterered on the routers for forthcoming reimage to bullseye T286911 |
[production] |
| 10:52 |
<mvolz@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'zotero' for release 'production' . |
[production] |
| 10:51 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host sretest1002.eqiad.wmnet |
[production] |
| 10:50 |
<marostegui> |
Upgrade db2102 db2116 db2130 db2145 db2146 |
[production] |
| 10:47 |
<mvolz@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'zotero' for release 'staging' . |
[production] |
| 10:27 |
<volans@cumin1001> |
START - Cookbook sre.experimental.reimage for host sretest1002.eqiad.wmnet |
[production] |
| 09:59 |
<jiji@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 09:55 |
<jiji@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 09:52 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for mx2002.wikimedia.org: Renew puppet certificate - jmm@cumin2002 |
[production] |
| 09:52 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.renew-cert for mx2002.wikimedia.org: Renew puppet certificate - jmm@cumin2002 |
[production] |
| 09:51 |
<jiji@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 09:40 |
<moritzm> |
reinstalling mx2002 (test server) to validate bullseye installs are fixed |
[production] |
| 09:31 |
<jiji@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 09:30 |
<jiji@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |