2151-2200 of 10000 results (23ms)
2021-03-02 ยง
21:59 <mforns@deploy1002> Started deploy [analytics/refinery@af99602]: Regular analytics weekly train [analytics/refinery@af99602101018664670a76d28cd755caf07dcde7] [production]
21:58 <mforns@deploy1002> deploy aborted: Regular analytics weekly train [analytics/refinery@COMMIT_HASH] (duration: 00m 01s) [production]
21:57 <mforns@deploy1002> Started deploy [analytics/refinery@af99602]: Regular analytics weekly train [analytics/refinery@COMMIT_HASH] [production]
21:51 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mwmaint2001.codfw.wmnet with reason: decom [production]
21:51 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on mwmaint2001.codfw.wmnet with reason: decom [production]
21:51 <legoktm> copied docker-registry package from stretch-wikimedia to buster-wikimedia (T272550) [production]
20:47 <krinkle@deploy1002> Synchronized wmf-config/InitialiseSettings.php: I7f387bf19e5f prep wgChronologyProtectorStash ahead of wmf.33 roll out to ensure cross-wiki consistency (duration: 01m 18s) [production]
20:04 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
20:03 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
20:00 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
20:00 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
19:56 <papaul> codfw mgmt is going down for 5 minutes for maintenance thank youn [production]
19:53 <kharlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
19:43 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: c48e40ac43445d6b038919133b951d6aaea960b7: Enable babel categorize on thwikisource (T275283) (duration: 01m 09s) [production]
19:42 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:39 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
19:28 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: f6fa5b3d2fa315518ce5d0995a653e038da05f24: Set local timezone for trwikivoyage to UTC (T275598) (duration: 01m 09s) [production]
19:15 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1002.eqiad.wmnet with reason: REIMAGE [production]
19:13 <volans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1002.eqiad.wmnet with reason: REIMAGE [production]
18:59 <ebernhardson> apply merge.policy.deletes_pct_allowed=20 to production-search-codfw commonswiki_file to encourage merging away deleted docs from T271493 [production]
18:53 <mholloway-shell@deploy1002> Synchronized php-1.36.0-wmf.32/extensions/EventLogging: Fix timestamp format for migrated events (T276235) (duration: 01m 10s) [production]
18:42 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1002.eqiad.wmnet with reason: REIMAGE [production]
18:40 <volans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1002.eqiad.wmnet with reason: REIMAGE [production]
18:28 <dduvall@deploy1002> Synchronized private/readme.php: Config: [[gerrit:667242|Extend wmfSwiftConfig placeholder keys]] (duration: 01m 09s) [production]
18:21 <mholloway-shell@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/EventLogging: Fix timestamp format for migrated events (T276235) (duration: 01m 09s) [production]
18:12 <vgutierrez> rolling restart of ats-tls on esams [production]
17:46 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@869a29b]: ores_bulk_ingest: Increase drafttopic error_threshold to 1 per 500 (duration: 02m 55s) [production]
17:43 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@869a29b]: ores_bulk_ingest: Increase drafttopic error_threshold to 1 per 500 [production]
17:39 <legoktm@deploy1002> Synchronized php-1.36.0-wmf.32/extensions/Graph/: Do not log graph errors to WMF servers (duration: 01m 08s) [production]
17:21 <legoktm@deploy1002> Synchronized php-1.36.0-wmf.32/extensions/ContentTranslation/: Re-apply: CX3 Build 0.1.0+20210223 (duration: 01m 10s) [production]
16:37 <jayme@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=eqiad [production]
16:37 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet [production]
16:33 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet [production]
16:14 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
16:13 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
16:10 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
16:10 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
15:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 100%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14563 and previous config saved to /var/cache/conftool/dbconfig/20210302-155932-root.json [production]
15:56 <jayme@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=helm-charts,name=eqiad [production]
15:56 <jayme@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=codfw [production]
15:55 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet [production]
15:52 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet [production]
15:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 85%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14562 and previous config saved to /var/cache/conftool/dbconfig/20210302-154429-root.json [production]
15:35 <tgr@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/GrowthExperiments/: Backport: [[gerrit:667811|HomepageHooks: Block search data hook if link recommendations are off (T276224)]] (duration: 01m 13s) [production]
15:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 75%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14561 and previous config saved to /var/cache/conftool/dbconfig/20210302-152925-root.json [production]
15:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 50%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14560 and previous config saved to /var/cache/conftool/dbconfig/20210302-151422-root.json [production]
15:00 <hnowlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
15:00 <hnowlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
14:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 25%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14559 and previous config saved to /var/cache/conftool/dbconfig/20210302-145918-root.json [production]
14:57 <vgutierrez> rolling restart of ats-tls on codfw [production]