3201-3250 of 10000 results (35ms)
2020-12-03 §
09:34 <moritzm> gnt-instance reboot ldap-replica2003 to validate new qemu [production]
09:14 <moritzm> installing qemu security updates on Stretch [production]
06:06 <marostegui> Create sockpuppet database on m2 T268505 [production]
04:13 <ejegg> updated fundraising CiviCRM from a2979cbba1 to 913ccdfd2b [production]
03:26 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
03:24 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
01:54 <eileen> process-control config revision is f863b32627 [production]
01:21 <mutante> lists1001 - remove "delete_held_messages" cronjob from root crontab - replaced by systemd timer - systemctl start delete_held_messages.service and confirmed it succeeded [production]
2020-12-02 §
23:42 <reedy@deploy1001> Synchronized php-1.36.0-wmf.20/includes/debug/logger/monolog/LogstashFormatter.php: T269286 (duration: 01m 07s) [production]
22:49 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
22:47 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
22:43 <twentyafterfour@deploy1001> Synchronized php-1.36.0-wmf.20/extensions/CategoryTree/: Deploying backport f6c2d74259b9 to wmf.20, bug: T269235 refs T263186 (duration: 01m 07s) [production]
22:38 <twentyafterfour@deploy1001> Synchronized php-1.36.0-wmf.20/includes/parser/: Deploying backports for wmf.20 refs T263186 (duration: 01m 08s) [production]
21:23 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
21:23 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
21:20 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
21:20 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
21:17 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
21:16 <clarakosi@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
21:14 <clarakosi@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
21:11 <clarakosi@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
21:03 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:01 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:56 <twentyafterfour> deploying backports for 1.36.0-wmf.20 refs T263186 [production]
20:52 <pt1979@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:50 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
20:49 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
20:49 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
20:42 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
20:41 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
20:38 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
20:32 <mbsantos@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
20:32 <mbsantos@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
20:27 <mbsantos@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
20:27 <mbsantos@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
20:25 <mbsantos@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
20:22 <mbsantos@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
20:21 <mbsantos@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
20:20 <mbsantos@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
20:14 <mutante> sodium - started update-ubuntu-mirror systemd timer - debugging why it fails; manually syncing with sudo -u mirror /usr/local/sbin/update-ubuntu-mirror [production]
20:10 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:08 <mutante> sodium systemctl reset-failed [production]
20:08 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
19:56 <mutante> sodium - systemctl restart update-tails-mirror.timer [production]
19:20 <mforns> restarted turnilo to clear deleted datasource [production]
19:17 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:643230|Add EventStream config for link recommendations (T261407)]] (duration: 01m 06s) [production]
18:22 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
18:20 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
18:20 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
18:20 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]