|
2021-06-30
§
|
| 13:26 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-fe2001.codfw.wmnet |
[production] |
| 13:26 |
<moritzm> |
installing fluidsynth security updates on stretch |
[production] |
| 13:25 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-fe1001.eqiad.wmnet |
[production] |
| 13:18 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-fe1001.eqiad.wmnet |
[production] |
| 13:18 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2004.codfw.wmnet |
[production] |
| 13:11 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2004.codfw.wmnet |
[production] |
| 13:04 |
<mutante> |
switching docker-registry to nginx light variant T164456 |
[production] |
| 13:00 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2003.codfw.wmnet |
[production] |
| 12:53 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2003.codfw.wmnet |
[production] |
| 12:52 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2002.codfw.wmnet |
[production] |
| 12:46 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2002.codfw.wmnet |
[production] |
| 12:45 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2001.codfw.wmnet |
[production] |
| 12:39 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2001.codfw.wmnet |
[production] |
| 12:35 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1004.eqiad.wmnet |
[production] |
| 12:29 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-be1004.eqiad.wmnet |
[production] |
| 12:29 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1002.eqiad.wmnet |
[production] |
| 12:24 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-be1002.eqiad.wmnet |
[production] |
| 12:19 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1001.eqiad.wmnet |
[production] |
| 12:17 |
<kart_> |
Updated cxserver to 2021-06-30-112813-production (T284900, T284885) |
[production] |
| 12:13 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-be1001.eqiad.wmnet |
[production] |
| 12:11 |
<kartik@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 12:06 |
<kartik@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 12:01 |
<kartik@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 11:46 |
<Lucas_WMDE> |
EU backport+config window done |
[production] |
| 11:46 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:701505|Remove $wmgWikibaseClientRepoConceptBaseUri (T257260)]] (2/2, beta) (disregard the earlier /3, I’m skipping the test file after all) (duration: 01m 04s) |
[production] |
| 11:44 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:701505|Remove $wmgWikibaseClientRepoConceptBaseUri (T257260)]] (1/3, prod) (duration: 01m 16s) |
[production] |
| 11:35 |
<moritzm> |
rolling restart of FPM/Apache on mw canaries to pick up gnutls/gcrypt security updates |
[production] |
| 11:11 |
<moritzm> |
installing libgcrypt security updates on buster |
[production] |
| 11:09 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@mwdebug2001:~$ sudo -u mwdeploy sh -c 'rm /srv/mediawiki/php-1.37.0-wmf.1/cache/l10n/l10n_cache-*.cdb && rmdir /srv/mediawiki/php-1.37.0-wmf.1/cache/l10n /srv/mediawiki/php-1.37.0-wmf.1/cache /srv/mediawiki/php-1.37.0-wmf.1' # clean up old l10n cache |
[production] |
| 11:08 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:701504|Stop setting Wikibase client repoConceptBaseUri (T257260)]] (duration: 01m 24s) |
[production] |
| 10:44 |
<moritzm> |
installing gnutls security updates on buster |
[production] |
| 10:31 |
<godog> |
add 200G to prometheus/eqiad for 'ops' instance |
[production] |
| 09:35 |
<godog> |
start swiftrepl-mw on ms-fe2005 post-switchover (credentials were missing) - T162123 |
[production] |
| 08:51 |
<jelto> |
jelto@puppetmaster1001:~$ sudo puppet cert -s gitlab2001.wikimedia.org # approve puppet certificate request for gitlab2001, fingerprint checked |
[production] |
| 08:47 |
<topranks> |
Removing BGP peers for AS48237 (Etihad Etisalat) and AS11404 (Wave Division Holdings) from cr2-eqiad (peers have left Equinix IX) |
[production] |
| 08:31 |
<godog> |
remove sdf1 from thanos-be1003 in swift - T285835 |
[production] |
| 07:43 |
<filippo@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host thanos-be1003.eqiad.wmnet |
[production] |
| 07:43 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-be1003.eqiad.wmnet |
[production] |
| 07:43 |
<filippo@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host thanos-be1003.eqiad.wmnet |
[production] |
| 07:37 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thanos-be1003.eqiad.wmnet |
[production] |
| 05:46 |
<ryankemper> |
[Cirrus] Unbanned `elastic2045`; now only `elastic2033` is banned in `codfw` |
[production] |
| 00:36 |
<tstarling@deploy1002> |
Synchronized wmf-config/db-labs.php: gerrit 701995 SQL query log (duration: 01m 05s) |
[production] |
| 00:35 |
<tstarling@deploy1002> |
Synchronized wmf-config/db-eqiad.php: gerrit 701995 SQL query log (duration: 01m 06s) |
[production] |
| 00:34 |
<tstarling@deploy1002> |
Synchronized wmf-config/db-codfw.php: gerrit 701995 SQL query log (duration: 01m 06s) |
[production] |
| 00:32 |
<tstarling@deploy1002> |
Synchronized wmf-config/CommonSettings.php: gerrit 701995 SQL query log (duration: 01m 05s) |
[production] |
| 00:31 |
<tstarling@deploy1002> |
Synchronized docroot/noc/db.php: gerrit 701995 SQL query log (duration: 01m 06s) |
[production] |
| 00:27 |
<tstarling@deploy1002> |
Synchronized wmf-config/logging.php: gerrit 701995 SQL query log (duration: 01m 15s) |
[production] |
| 00:01 |
<urbanecm> |
(following up previous SAL item) TrainBranchBot was removed from wmf-deployment group because of T285819 |
[production] |