51-100 of 10000 results (20ms)
2020-05-28 §
08:46 <XioNoX> deactivate peering/transit on cr2-eqord - T243080 [production]
08:45 <XioNoX> de-pref all OSPF links to cr2-eqord - T243080 [production]
08:13 <marostegui> Pool db1141 into labsdb analytics role - T249188 [production]
07:33 <gilles@deploy1001> Synchronized static/images: T252108 Deploying optimised static PNGs (duration: 01m 39s) [production]
07:31 <gilles@deploy1001> Synchronized static/apple-touch: T252108 Deploying optimised static PNGs (duration: 01m 12s) [production]
06:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1081 from API and set its weight to 0 on main traffic - preparation for tomorrow's failover T253808', diff saved to https://phabricator.wikimedia.org/P11329 and previous config saved to /var/cache/conftool/dbconfig/20200528-063037-marostegui.json [production]
04:44 <marostegui> Run check_private data on db1141 - T249188 [production]
04:22 <marostegui> Stop MySQL on db1141 - T249188 [production]
2020-05-27 §
23:20 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Add autoreviewrestore right to rollbacker group on hiwiki (T252986) (duration: 01m 05s) [production]
23:16 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Add thwiki Draft namespace to wmgExemptFromUserRobotsControlExtra and enable VE there (T252959) (duration: 01m 06s) [production]
22:58 <gehel@cumin1001> END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) [production]
22:02 <crusnov@deploy1001> Finished deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part4) (duration: 00m 10s) [production]
22:02 <crusnov@deploy1001> Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part4) [production]
22:01 <crusnov@deploy1001> Finished deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part3) (duration: 01m 29s) [production]
22:00 <crusnov@deploy1001> Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part3) [production]
22:00 <crusnov@deploy1001> deploy aborted: Netbox Upgrade to 2.8.4 (part2) (duration: 01m 31s) [production]
21:58 <crusnov@deploy1001> Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part2) [production]
21:58 <crusnov@deploy1001> Finished deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.1 (part1) (duration: 01m 01s) [production]
21:57 <crusnov@deploy1001> Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.1 (part1) [production]
20:43 <gehel@cumin1001> START - Cookbook sre.postgresql.postgres-init [production]
20:28 <marostegui> Decrease innodb poolsize on s4 master and restart mysql [production]
20:11 <mbsantos@deploy1001> Finished deploy [mobileapps/deploy@9dc827f]: Update mobileapps to b3b9214c (T253648) (duration: 03m 31s) [production]
20:08 <mbsantos@deploy1001> Started deploy [mobileapps/deploy@9dc827f]: Update mobileapps to b3b9214c (T253648) [production]
20:04 <twentyafterfour@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.32 refs T253022 (duration: 01m 04s) [production]
20:03 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.32 refs T253022 [production]
20:00 <gehel@cumin1001> END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) [production]
19:56 <twentyafterfour@deploy1001> scap failed: average error rate on 4/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details) [production]
19:46 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.34/includes/parser/CoreParserFunctions.php: T253725 Partially revert 'Fix impedance mismatch with Parser::getRevisionRecordObject()' (duration: 01m 05s) [production]
19:12 <joal@deploy1001> Finished deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train (an-launcher1001 only) [8a3dcb3] (duration: 06m 07s) [production]
19:09 <jforrester@deploy1001> Synchronized dblists/mobilemainpagelegacy.dblist: T32405 Stop special casing the main page on mobile for twelve wikis (duration: 01m 05s) [production]
19:06 <joal@deploy1001> Started deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train (an-launcher1001 only) [8a3dcb3] [production]
19:03 <joal@deploy1001> Finished deploy [analytics/refinery@8a3dcb3] (thin): Analytics regular weekly train THIN [8a3dcb3] (duration: 00m 08s) [production]
19:03 <joal@deploy1001> Started deploy [analytics/refinery@8a3dcb3] (thin): Analytics regular weekly train THIN [8a3dcb3] [production]
19:03 <joal@deploy1001> Finished deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train [8a3dcb3] (duration: 21m 20s) [production]
18:41 <joal@deploy1001> Started deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train [8a3dcb3] [production]
18:06 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable DiscussionTools as beta on mediawiki.org, part II T251208 (duration: 01m 05s) [production]
17:56 <jayme> updated tiller to 2.16.7-wmf1 for all services in kubernetes cluster: eqiad [production]
17:53 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable DiscussionTools as beta on mediawiki.org T251208 (duration: 01m 05s) [production]
17:42 <gehel@cumin1001> START - Cookbook sre.postgresql.postgres-init [production]
17:40 <gehel> repool maps2003 [production]
17:32 <twentyafterfour@deploy1001> Synchronized php-1.35.0-wmf.34/extensions/Translate/: Deploy https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Translate/+/599027/ to wmf.34 refs T253748 and T253022 (duration: 01m 07s) [production]
16:55 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
16:53 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
16:52 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
16:26 <hnowlan@deploy1001> Finished deploy [cpjobqueue/deploy@c8c653e]: Disabling ThumbnailRender as a test of k8s cpjobqueue (duration: 01m 57s) [production]
16:24 <hnowlan@deploy1001> Started deploy [cpjobqueue/deploy@c8c653e]: Disabling ThumbnailRender as a test of k8s cpjobqueue [production]
16:10 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
16:09 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
16:06 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:52 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Stop defining wmgUsePerformanceInspector, unread T253689 (duration: 01m 04s) [production]