3151-3200 of 10000 results (42ms)
2020-10-26 §
13:21 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:00 <hashar> Upgrading docker-ce to 19.03.13 and containerd to 1.3.7 on integration-agent-docker-1001, integration-agent-docker-1002, integration-agent-docker-1003 and integration-agent-docker-1004 # T265615 [releng]
11:52 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: bff6b37a55fe8f260fe00cbb942c53101167fb07: Add foto.digitalarkivet.no to wgCopyUploadsDomains whitelist of Wikimedia Commons (T266390) (duration: 01m 14s) [production]
11:27 <elukey@cumin1001> END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) [production]
11:27 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
11:26 <elukey@cumin1001> END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) [production]
11:26 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
11:12 <hashar> Upgrading containerd.io from 1.2.10 to 1.2.13-2 on integration-agent-docker-1005, integration-agent-docker-1006, integration-agent-docker-1007, integration-agent-docker-1008 # T265615 [releng]
11:11 <vgutierrez> upgrade trafficserver to 8.0.8-1wm3 on cp4032 - T265911 [production]
11:03 <hashar> Bring back integration-agent-docker-1020 . It is not the only one affected by T265615 which probably rules out Ceph as the slowness root cause (T260916) [releng]
11:02 <elukey@cumin1001> END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) [production]
11:02 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
10:56 <hashar> Upgrading containerd.io from 1.2.10 to 1.2.13-2 on integration-agent-docker-1001, integration-agent-docker-1002, integration-agent-docker-1003, integration-agent-docker-1004 # T265615 [releng]
10:51 <vgutierrez> manually reloading nginx on cloudelastic[1005-1006] [production]
10:29 <vgutierrez> upload trafficserver 8.0.8-1wm3 to apt.wm.org (buster) - T265911 [production]
10:18 <godog> roll restart pybal to apply latest configuration [production]
09:51 <jayme> published docker-registry.discovery.wmnet/eventrouter:0.3.0-3 [production]
09:31 <moritzm> restarting PHP FPM on mw canaries to pick up freetype update [production]
09:04 <godog> swift codfw-prod: bump object weight for ms-be2057 - T261633 [production]
08:58 <moritzm> installing freetype security updates for stretch [production]
08:57 <XioNoX> remove down sessions to AS38758 [production]
08:51 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:51 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:43 <XioNoX> remove down sessions to AS8560 [production]
08:41 <XioNoX> remove down sessions to AS31334 [production]
08:28 <XioNoX> remove down sessions to AS6327 [production]
08:27 <XioNoX> remove down sessions to AS8674 [production]
08:25 <XioNoX> remove down sessions to AS24429 [production]
08:21 <XioNoX> remove down sessions to AS16509 [production]
06:59 <_joe_> rolling restart of php7.2-fpm on the codfw jobrunners, to reduce the number of dangling transcodes after restarting cp-jobqueue for a deploy [production]
06:59 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
06:16 <oblivian@cumin2001> conftool action : set/pooled=no; selector: cluster=jobrunner,dc=codfw,name=mw224.* [production]
06:15 <oblivian@cumin2001> conftool action : set/pooled=no; selector: cluster=videoscaler,dc=codfw,name=mw228.* [production]
06:10 <marostegui> Warm up tables T261914 [production]
02:54 <Reedy> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/636229 T266424 [releng]
2020-10-25 §
21:46 <wm-bot> <lucaswerkmeister> deployed bd4c445f02 (edit mode fix) [tools.lexeme-forms]
21:11 <wm-bot> <lucaswerkmeister> deployed 782dfdabee (fixes for edit mode and ordia links) [tools.lexeme-forms]
16:20 <andrewbogott> adding cloudvirt1038 to the 'ceph' aggregate and removing from the 'spare' aggregate. We need this space while waiting on network upgrades for empty cloudvirts (T216195) [admin]
15:53 <dwisehaupt> kernel upgrade and reboot for frdb1003 [production]
15:50 <dwisehaupt> kernel upgrade and reboot for fran1001 [production]
2020-10-24 §
13:47 <wm-bot> <lucaswerkmeister> deployed 792db2a9f9 (edit mode language_code parameter) [tools.lexeme-forms]
2020-10-23 §
22:56 <mutante> added Nuria to "nda" LDAP group - leaving her in "wmf" until the actual last day - shell account remains so no puppet change needed in ldap_only_admins (T266086) [production]
22:22 <legoktm> imported pack_0.14.2-1_amd64.deb into buster-tools (T266270) [tools]
15:42 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:37 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
13:04 <ema> rolling thumbor-instances restart to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/636012/ T266155 [production]
12:47 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . [production]
11:30 <arturo> [codfw1dev] openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record nat.cloudgw.codfw1dev.wikimediacloud.org. --description "created by hand" 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa. (T261724) [admin]