401-450 of 10000 results (36ms)
2021-10-28 ยง
23:50 <robh@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4034.ulsfo.wmnet with OS buster [production]
23:46 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host cp4034.ulsfo.wmnet with OS buster [production]
23:43 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host cp4033.ulsfo.wmnet with OS buster [production]
23:33 <brennen> end of UTC late backport & config window [production]
23:28 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:733403|Add User and User talk to $wgExemptFromUserRobotsControl on zhwiki (T288947)]] (duration: 00m 56s) [production]
23:21 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:16 <mutante> mwdebug1001 - letting puppet remove all mediawiki font packages using new Hiera key 'profile::mediawiki::webserver::install_fonts: false' to make sure we really don't need them (T294378) [production]
22:06 <mutante> releases1002 - releases2002, something broke puppet here about 27.5 hours ago. lookup() did not find a value for the name 'profile::docker::storage::physical_volumes' [production]
22:03 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:01 <mutante> re-enabled puppet on deploy*, parse* and thenr everything else [production]
21:59 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
21:56 <mutante> re-enabled puppet on mw-app-canary, mwmaint, labweb1002,.. [production]
21:53 <mutante> re-enabled puppet on mw-api-canary [production]
21:48 <arnoldokoth> restore script scheduled to run on gitlab2001 (T285867) [production]
21:46 <arnoldokoth> restoreccccccvkvhgbvtklgce kkbeuvvuskljihickdbgcunljcr scheduled to run on gitlab2001 (T285867) [production]
21:42 <mutante> sudo cumin 'C:profile::mediawiki::common' "disable-puppet 'gerrit:734798 - ${USER}'" [production]
20:37 <mutante> ensured gitlab restore timer is running only on passive server and re-enabled it - https://gerrit.wikimedia.org/r/c/operations/puppet/+/735437 T274463 [production]
20:30 <mutante> gitlab2001 - re-enabled gitlab-restore-from-backup service [production]
20:19 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:16 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:13 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: c23e7ab0f88a64b8f656e06949518fb816b2dd56: foundationwiki: Set wmgLocalAuthLoginOnly=false temporarily (T205347) (duration: 00m 55s) [production]
20:06 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:03 <twentyafterfour@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.38.0-wmf.6 refs T293947 [production]
20:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:53 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:52 <twentyafterfour@deploy1002> Synchronized php: group1 wikis to 1.38.0-wmf.6 refs T293947 (duration: 01m 02s) [production]
19:51 <twentyafterfour@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.6 refs T293947 [production]
19:50 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:43 <legoktm@deploy1002> Synchronized php-1.38.0-wmf.6/includes/libs/http/MultiHttpClient.php: Hack: temporarily log headers in MultiHttpClient (duration: 01m 02s) [production]
19:41 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:40 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:40 <legoktm@deploy1002> Synchronized php-1.38.0-wmf.5/includes/libs/http/MultiHttpClient.php: Hack: temporarily log headers in MultiHttpClient (duration: 01m 02s) [production]
19:37 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
19:37 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:36 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.6/includes/GlobalFunctions.php: b517ebd29396c23f66806cb0dca1d1b330c6e5be: Revert "wfParseUrl: rely on parse_url for proto-relative urls" (T294559) (duration: 01m 03s) [production]
19:29 <mutante> [netflow4001:~] $ sudo systemctl reset-failed [production]
19:23 <legoktm> manually restarted nginx on ncredir4002 after accidental ganeti reboot [production]
19:23 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:14 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.6/extensions/FlaggedRevs/: 2cd2a4e62af1b66ad29adf27d761129e8fea388a: REST: Avoid making wpaccuracy required in API requests (T294544) (duration: 01m 03s) [production]
19:12 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 60413dcece4a9e5ec125be49315dc22ec3b85cc7: foundationwiki: Use shared OAuth tables (T205347) (duration: 01m 04s) [production]
19:10 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:57 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:54 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:53 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 4e0200e3f5ad9c7514ddbeda6f7ee4f8b5ed2ec7: emailuser ratelimit: Use user-global rather than user (T293866) (duration: 01m 04s) [production]
17:23 <cstone> civicrm revision changed from dba74c443b to 403be9ce05 [production]
16:53 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:50 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]