801-850 of 10000 results (25ms)
2020-07-28 Β§
19:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1142', diff saved to https://phabricator.wikimedia.org/P12097 and previous config saved to /var/cache/conftool/dbconfig/20200728-191926-marostegui.json [production]
19:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1147', diff saved to https://phabricator.wikimedia.org/P12096 and previous config saved to /var/cache/conftool/dbconfig/20200728-191237-marostegui.json [production]
19:11 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@69bbbbb]: airflow: drop_old_data_daily: top_queries table renamed to fulltext_head_queries (duration: 00m 53s) [production]
19:11 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@69bbbbb]: airflow: drop_old_data_daily: top_queries table renamed to fulltext_head_queries [production]
19:09 <jhuneidi@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
19:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12095 and previous config saved to /var/cache/conftool/dbconfig/20200728-190933-marostegui.json [production]
19:06 <jhuneidi@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
19:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12094 and previous config saved to /var/cache/conftool/dbconfig/20200728-190517-marostegui.json [production]
19:03 <jhuneidi@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
19:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1147', diff saved to https://phabricator.wikimedia.org/P12093 and previous config saved to /var/cache/conftool/dbconfig/20200728-190137-marostegui.json [production]
18:35 <cdanis> βœ”οΈ cdanis@lvs1015.eqiad.wmnet ~ πŸ•β˜• sudo ipvsadm -D -t 10.2.2.51:9283 [production]
18:29 <cdanis> ❌cdanis@lvs1016.eqiad.wmnet ~ πŸ•β˜• sudo ipvsadm -D -t 10.2.2.51:9283 [production]
18:29 <catrope@deploy1001> Synchronized php-1.36.0-wmf.2/extensions/GrowthExperiments/extension.json: Fix reference to MentorChangeLogFormatter (T259041) (duration: 01m 05s) [production]
18:20 <catrope@deploy1001> Synchronized wmf-config/CommonSettings.php: No-op sync for wmgUseWikimediaApiPortal and wmgUseWikimediaApiPortalOAuth (2 of 2) (duration: 00m 58s) [production]
18:17 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: No-op sync for wmgUseWikimediaApiPortal and wmgUseWikimediaApiPortalOAuth (1 of 2) (duration: 01m 05s) [production]
18:16 <cdanis> primary pybal restart βœ”οΈ cdanis@lvs1015.eqiad.wmnet ~ πŸ•‘β˜• sudo systemctl restart pybal.service [production]
18:14 <cdanis> backup pybal restart: βœ”οΈ cdanis@lvs1016.eqiad.wmnet ~ πŸ•‘β˜• sudo systemctl restart pybal.service [production]
18:10 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:05 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
18:05 <catrope@deploy1001> Synchronized php-1.36.0-wmf.2/includes/libs/filebackend/SwiftFileBackend.php: Fix index error in SwiftFileBackend (T259023) (duration: 01m 07s) [production]
17:46 <volker-e@deploy1001> Finished deploy [design/style-guide@e3fda83]: Deploy design/style-guide: (duration: 00m 05s) [production]
17:46 <volker-e@deploy1001> Started deploy [design/style-guide@e3fda83]: Deploy design/style-guide: [production]
17:41 <volans> run apt-get clean on wtp[1046,1048].eqiad.wmnet and wtp2001.codfw.wmnet to free ~`2GB as they were 100% - T258775 [production]
17:33 <XioNoX> standardize mr1-esams interfaces [production]
17:30 <brennen@deploy1001> sync aborted: (no justification provided) (duration: 28m 53s) [production]
17:03 <brennen> prior scap sync for https://gerrit.wikimedia.org/r/c/mediawiki/core/+/616842 (T259023) [production]
17:02 <brennen@deploy1001> Started scap: (no justification provided) [production]
16:51 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@0982d4e]: convert_to_esbulk: repair variable ref before assign (duration: 04m 33s) [production]
16:49 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:47 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:47 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@0982d4e]: convert_to_esbulk: repair variable ref before assign [production]
16:45 <XioNoX> remove mr1-codfw source NAT (not used) [production]
16:43 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1045.eqiad.wmnet [production]
16:39 <ppchelko@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
16:36 <ppchelko@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
16:34 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
16:33 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet [production]
16:32 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet [production]
16:31 <XioNoX> mr1-eqiad# delete security nat source rule-set mgmt-to-untrust (unused, no matching ACL) [production]
16:25 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:23 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:21 <hnowlan> imported envoyproxy 1.15.0-1 deb into component/envoy-future for buster-wikimedia [production]
16:11 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1042.eqiad.wmnet [production]
16:09 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1043.eqiad.wmnet [production]
15:54 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:52 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:51 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:50 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:48 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:48 <jayme@cumin1001> START - Cookbook sre.hosts.downtime [production]