551-600 of 10000 results (59ms)
2020-04-10 §
08:03 <hashar@deploy1001> Finished deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) (duration: 00m 05s) [production]
08:03 <hashar@deploy1001> Started deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) [production]
07:52 <mutante> closing port 80 on phab hosts for caching servers [production]
07:37 <ema> cp3050: back to vhtcpd for the holidays T249583 [production]
07:00 <mutante> sodium - sudo -u mirror ftpsync [production]
06:58 <mutante> armed keyholder on deploy1001 [production]
06:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:15 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:00 <marostegui> Stop MySQL on pc1008 for upgrade [production]
2020-04-09 §
23:44 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 58s) [production]
23:27 <catrope@deploy1001> Synchronized wmf-config/mobile.php: Drop fallback support for wgMobileFrontendLogo (T248500) (duration: 00m 58s) [production]
23:21 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Drop unused config for main page CSS (T243996) (duration: 00m 58s) [production]
23:17 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Add extendedconfirmed group and protection level on jawiki (T249820) (duration: 00m 59s) [production]
22:01 <sukhe> running initial metadb sync on cescout1001 [production]
19:43 <mholloway-shell@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
19:41 <mholloway-shell@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
19:39 <mholloway-shell@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . [production]
19:08 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.27 refs T247774 [production]
19:01 <longma> deploying 1.35.0-wmf.27 to all wikis [production]
17:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:50 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:50 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:40 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
17:24 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
17:18 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
14:39 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
14:32 <XioNoX> disable down interfaces from fasw-c-codfw (mintaka) [production]
13:45 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
13:31 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
12:43 <mlitn@deploy1001> Synchronized php-1.35.0-wmf.27/extensions/MachineVision/: [MachineVision] Fix statement creation from suggestion (duration: 01m 09s) [production]
12:31 <ema> cp3051: upgrade varnish to 5.1.3-1wm13 once again, restart varnish-fe T249809 [production]
11:57 <XioNoX> offload more traffic from NTT eqiad - T249808 [production]
11:20 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|587257|Enable ContentTranslation as a default tool in Slovenian WP (T248836)]], take II (duration: 01m 06s) [production]
11:19 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|587257|Enable ContentTranslation as a default tool in Slovenian WP (T248836)]] (duration: 01m 07s) [production]
10:50 <vgutierrez> rolling upgrade to trafficserver 8.0.6-1mw7 [production]
10:50 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:50 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:49 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
10:49 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:43 <ema> repool cp3051 T249809 [production]
10:30 <ema> cp3051: re-enable transient storage limit, downgrade varnish to 5.1.3-1wm12 (no 0035-vbf_stp_condfetch_crash.patch) and restart varnish-fe T249809 [production]
09:46 <ema> cp3051: disable transient storage limit and restart varnish-fe T249809 [production]
09:31 <XioNoX> offload traffic from NTT eqiad - T249808 [production]
07:56 <mutante> contint2001 - a2dismod mpm_event - then run puppet to let it enable php_mod_7.3 (race condition like mentioned in https://gerrit.wikimedia.org/r/c/operations/puppet/+/451206) (T224591) [production]
07:56 <mutante> contint2001 - a2dismod mpm_event - then run puppet to let it enable php_mod_7.3 (race condition like mentioned in https://gerrit.wikimedia.org/r/c/operations/puppet/+/451206) [production]
07:24 <moritzm> synched jenkins 222.1 to apt.wikimedia.org (buster-wikimedia, thirdparty/ci) T224591 [production]
07:12 <marostegui> Repool labsdb1011 [production]
07:10 <XioNoX> switch urpf from log to syslog in ulsfo [production]
07:04 <XioNoX> re-activate BGP to Zayo in eqiad [production]