5001-5050 of 10000 results (82ms)
2022-05-16 §
07:18 <dcausse> restarting blazegraph on wdqs1007 (BlazegraphFreeAllocatorsDecreasingRapidly) [production]
2022-05-15 §
21:47 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 07s) [production]
21:46 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
21:42 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 07s) [production]
21:42 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
21:39 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 08s) [production]
21:39 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
21:30 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 08s) [production]
21:30 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
21:14 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 08s) [production]
21:14 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
2022-05-14 §
08:34 <jynus@cumin1001> dbctl commit (dc=all): 'Depool db1172', diff saved to https://phabricator.wikimedia.org/P27830 and previous config saved to /var/cache/conftool/dbconfig/20220514-083421-jynus.json [production]
00:53 <razzi@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on an-tool1005.eqiad.wmnet with reason: Server need to be downgraded to stretch, on monday [production]
00:53 <razzi@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on an-tool1005.eqiad.wmnet with reason: Server need to be downgraded to stretch, on monday [production]
2022-05-13 §
23:42 <razzi@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-tool1007.eqiad.wmnet with reason: Upgrade turnilo [production]
23:42 <razzi@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-tool1007.eqiad.wmnet with reason: Upgrade turnilo [production]
23:14 <razzi@deploy1002> Finished deploy [analytics/turnilo/deploy@bf60521]: Staging deployment of turnilo 1.35 (duration: 00m 08s) [production]
23:13 <razzi@deploy1002> Started deploy [analytics/turnilo/deploy@bf60521]: Staging deployment of turnilo 1.35 [production]
17:37 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices1003.wikimedia.org [production]
17:31 <andrew@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudservices1003.wikimedia.org [production]
17:30 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices1004.wikimedia.org [production]
17:24 <andrew@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudservices1004.wikimedia.org [production]
17:24 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudservices1004.wikimedia.org [production]
17:24 <andrew@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudservices1004.wikimedia.org [production]
15:57 <_joe_> uploading conftool 2.2.0 to buster, bullseye T305824 T305582 T305607 T305638 T307905 T308100 [production]
12:38 <akosiaris@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
12:38 <akosiaris@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply [production]
12:37 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
12:37 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [production]
12:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2140 after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P27824 and previous config saved to /var/cache/conftool/dbconfig/20220513-121832-marostegui.json [production]
12:09 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
11:59 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [production]
11:57 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
11:47 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [production]
11:40 <moritzm> installing idp-test1002 T308214 [production]
10:55 <moritzm> installing idp-test2002 T308214 [production]
10:41 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on ganeti4002.ulsfo.wmnet with reason: Remove from cluster for eventual reimage [production]
10:41 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on ganeti4002.ulsfo.wmnet with reason: Remove from cluster for eventual reimage [production]
10:18 <vgutierrez> disable puppet on gerrit1001 to fix /etc/ssh/ssh_config [production]
08:39 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
08:03 <jynus> moving s2 database from db2101 to db2097 T299920 [production]
07:59 <moritzm> draining ganeti4002 T307997 [production]
07:52 <XioNoX> add init7 transit in drmrs [production]
07:39 <root@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti4001.ulsfo.wmnet to ganeti01.svc.ulsfo.wmnet [production]
07:39 <root@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti4001.ulsfo.wmnet to ganeti01.svc.ulsfo.wmnet [production]
07:27 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4001.ulsfo.wmnet [production]
07:20 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti4001.ulsfo.wmnet [production]
07:18 <Amir1> start of mwscript extensions/Echo/maintenance/removeOrphanedEvents.php --wiki=wikidatawiki --force (T308084) [production]
02:14 <ejegg> updated payments-wiki from 8f46af9d to 590fac28 [production]
2022-05-12 §
21:56 <razzi@deploy1002> Finished deploy [analytics/turnilo/deploy@a2bdc3e]: (no justification provided) (duration: 02m 08s) [production]