6351-6400 of 10000 results (34ms)
2020-12-14 §
20:09 <jiji@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1031.eqiad.wmnet with reason: REIMAGE [production]
19:50 <effie> disable puppet on mc1031, mc2031 to install buster [production]
19:45 <mutante> mwdebug1003 - removing zero.wikimedia.org include for testing [production]
19:43 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 3b5974ff7f57d19732cd1e7f7f492b778daf6cfc: zhwikinews: Grant suppressredirect to autoconfirmed (T270023) (duration: 00m 55s) [production]
19:34 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cf36ad6e89acd71ca0bc985eb5399fecec64fc5f: hrwiki: Add draft namespace (T268740) (duration: 00m 56s) [production]
19:31 <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: gerrit:649359 group1: Enable OldRevisionParserCache (duration: 00m 55s) [production]
19:28 <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: gerrit:644317 Remove wgParserCacheUseJson setting (duration: 00m 56s) [production]
19:24 <ppchelko@deploy1001> Synchronized php-1.36.0-wmf.21/extensions/Popups: Backport gerrit:649408 Revert Remove title attributes at init (duration: 00m 59s) [production]
18:25 <ryankemper> T269204 Restarting `wdqs-blazegraph` prometheus exporter across all wdqs instances:`sudo cumin -b 12 'P{wdqs*}' 'sudo systemctl restart prometheus-blazegraph-exporter-wdqs-blazegraph.service'` [production]
18:05 <hnowlan@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=mw1265.eqiad.wmnet [production]
18:04 <crusnov@deploy1001> Finished deploy [netbox/deploy@2fc439e]: Redeploy Netbox 2.8 to netbox-next T266488 p2 (duration: 00m 05s) [production]
18:04 <crusnov@deploy1001> Started deploy [netbox/deploy@2fc439e]: Redeploy Netbox 2.8 to netbox-next T266488 p2 [production]
18:04 <crusnov@deploy1001> Finished deploy [netbox/deploy@2fc439e]: Redeploy Netbox 2.8 to netbox-next T266488 p1 (duration: 00m 33s) [production]
18:03 <crusnov@deploy1001> Started deploy [netbox/deploy@2fc439e]: Redeploy Netbox 2.8 to netbox-next T266488 p1 [production]
17:59 <hnowlan> depooled mw1265 for reimaging [production]
16:55 <jayme@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
16:55 <jayme@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
14:57 <root@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:54 <root@cumin1001> START - Cookbook sre.dns.netbox [production]
14:52 <root@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
14:46 <root@cumin1001> START - Cookbook sre.dns.netbox [production]
14:09 <jbond42> upload modified golang-cfssl to apt [production]
13:45 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:649321|Enable Wikibase Repo ID generator logging on Wikidata (T268625)]] (duration: 00m 55s) [production]
12:55 <Lucas_WMDE> EU backport+config window done [production]
12:54 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:644564|Enable Wikibase Repo ID generator logging on Test Wikidata (T268625)]] (2/2) (duration: 00m 54s) [production]
12:53 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:644564|Enable Wikibase Repo ID generator logging on Test Wikidata (T268625)]] (1/2) (duration: 00m 54s) [production]
12:45 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:643874|Add log channel Wikibase.IdGenerator (T268625)]] (Beta-only sync to avoid drift) (duration: 00m 55s) [production]
12:43 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:643874|Add log channel Wikibase.IdGenerator (T268625)]] (duration: 00m 54s) [production]
12:39 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:649307|Enable QuickSurveys on commonswiki (T258419)]] (duration: 00m 55s) [production]
12:09 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:640934|Add Media Search survey (T258419)]] (duration: 00m 55s) [production]
11:34 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:649304| Bumping portals to master (T128546)]] (duration: 00m 54s) [production]
11:33 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:649304| Bumping portals to master (T128546)]] (duration: 00m 56s) [production]
10:34 <godog> add 100G to prometheus 'global' in codfw [production]
10:32 <akosiaris> Adding kubernetes codfw staging cluster configuration to cr*-codfw [production]
10:17 <marostegui> Stop mysql on db2131 to clone db2142 [production]
10:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2131 to clone db2142', diff saved to https://phabricator.wikimedia.org/P13542 and previous config saved to /var/cache/conftool/dbconfig/20201214-101611-marostegui.json [production]
10:12 <ladsgroup@deploy1001> Synchronized php-1.36.0-wmf.21/extensions/Wikibase/client/includes: [[gerrit:648283|Avoid loading the whole item in every client page view (T269960)]] (duration: 00m 25s) [production]
10:03 <ladsgroup@deploy1001> Scap failed!: 4/9 canaries failed their endpoint checks(https://en.wikipedia.org) [production]
09:51 <godog> swift codfw-prod: more weight to ms-be20[58-61] - T269337 [production]
09:45 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on cloudvirt1024.eqiad.wmnet with reason: T269419 [production]
09:45 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 6 days, 0:00:00 on cloudvirt1024.eqiad.wmnet with reason: T269419 [production]
08:40 <godog> swift eqiad-prod: add weight to ms-be106[0-3] - T268435 [production]
2020-12-11 §
22:05 <dduvall@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
22:02 <dduvall@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
21:59 <dduvall@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
21:57 <akosiaris> add docker-ce_18.06.3~ce~3-0~debian_amd64.deb to apt.wikimedia.org stretch-wikimedia/thirdparty/k8s [production]
21:46 <Amir1> Running schema changes on wikitech database for T269348 [production]
21:45 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
21:42 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
21:41 <dduvall@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]