1901-1950 of 10000 results (21ms)
2020-08-10 §
09:49 <jayme@cumin1001> START - Cookbook sre.discovery.depool [production]
09:21 <marostegui> Promote dbproxy1019 back T255408 [production]
08:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:21 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:43 <marostegui> Remove revision triggers from db2094:3318 T238966 [production]
06:42 <marostegui> Stop replication on s8 codfw master to deploy MCR change, this will generate lag on s8 codfw T238966 [production]
04:46 <marostegui> Depool dbproxy1019 for reimage T255408 [production]
2020-08-09 §
21:58 <ejegg> updated payments-wiki from cd012f37f1 to 932aacde54 [production]
03:53 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
2020-08-08 §
02:23 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
02:21 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
02:19 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
2020-08-07 §
16:42 <jforrester@deploy1001> Synchronized php-1.36.0-wmf.3/extensions/DiscussionTools/: T259855 Revert new reply API (duration: 01m 06s) [production]
15:01 <volans> import DNS names for network devices in Netbox - T258729 [production]
13:27 <godog> bounce pybal on lvs1016 and then lvs1015 to reset state, logstash1025 reported down but actually up [production]
10:27 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:27 <sukhe@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:02 <elukey> reboot deneb via ganeti2021 (hostname config pointing to recdns for some reason) [production]
09:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1092', diff saved to https://phabricator.wikimedia.org/P12195 and previous config saved to /var/cache/conftool/dbconfig/20200807-091527-marostegui.json [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P12194 and previous config saved to /var/cache/conftool/dbconfig/20200807-084747-marostegui.json [production]
08:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P12193 and previous config saved to /var/cache/conftool/dbconfig/20200807-080719-marostegui.json [production]
07:50 <godog> prometheus codfw lvextend --resize --size +60G /dev/mapper/vg--hdd-prometheus--global [production]
07:49 <godog> prometheus codfw lvextend --resize --size +30G /dev/mapper/vg--ssd-prometheus--k8s [production]
07:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P12192 and previous config saved to /var/cache/conftool/dbconfig/20200807-074658-marostegui.json [production]
06:53 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 for upgrade', diff saved to https://phabricator.wikimedia.org/P12191 and previous config saved to /var/cache/conftool/dbconfig/20200807-063431-marostegui.json [production]
2020-08-06 §
23:21 <catrope@deploy1001> Synchronized php-1.36.0-wmf.3/extensions/GrowthExperiments/: Fixes for WelcomeSurvey language question (T232410) (duration: 00m 59s) [production]
23:04 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Change GrowthExperiments mentor list on fawiki (T253291) (duration: 00m 59s) [production]
21:43 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:41 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:40 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:39 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
21:39 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:35 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
21:33 <brennen@deploy1001> Synchronized php-1.36.0-wmf.3/vendor: [[gerrit:618850|Update git submodules (vendor)]] (T259832) (duration: 01m 08s) [production]
21:32 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
20:51 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
20:51 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
20:47 <shdubsh> restart logstash -- pipeline appears stuck [production]
20:38 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:38 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:19 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:19 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:19 <brennen> manually updating the vendor submodule on 1.36.0 for T259832 [production]
20:15 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
20:15 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
19:48 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
19:47 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
19:45 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - wgEventStreams - fix another typo in eventgate stream config - T251935 (duration: 00m 58s) [production]