4851-4900 of 10000 results (67ms)
2022-05-16 §
07:39 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:39 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:38 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:36 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 67ce6ce: zhwikisource: Add NS100 to wgNamespacesToBeSearchedDefault (T308393) (duration: 00m 50s) [production]
07:18 <dcausse> restarting blazegraph on wdqs1007 (BlazegraphFreeAllocatorsDecreasingRapidly) [production]
2022-05-15 §
21:47 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 07s) [production]
21:46 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
21:42 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 07s) [production]
21:42 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
21:39 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 08s) [production]
21:39 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
21:30 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 08s) [production]
21:30 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
21:14 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 08s) [production]
21:14 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) [production]
2022-05-14 §
08:34 <jynus@cumin1001> dbctl commit (dc=all): 'Depool db1172', diff saved to https://phabricator.wikimedia.org/P27830 and previous config saved to /var/cache/conftool/dbconfig/20220514-083421-jynus.json [production]
00:53 <razzi@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on an-tool1005.eqiad.wmnet with reason: Server need to be downgraded to stretch, on monday [production]
00:53 <razzi@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on an-tool1005.eqiad.wmnet with reason: Server need to be downgraded to stretch, on monday [production]
2022-05-13 §
23:42 <razzi@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-tool1007.eqiad.wmnet with reason: Upgrade turnilo [production]
23:42 <razzi@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-tool1007.eqiad.wmnet with reason: Upgrade turnilo [production]
23:14 <razzi@deploy1002> Finished deploy [analytics/turnilo/deploy@bf60521]: Staging deployment of turnilo 1.35 (duration: 00m 08s) [production]
23:13 <razzi@deploy1002> Started deploy [analytics/turnilo/deploy@bf60521]: Staging deployment of turnilo 1.35 [production]
17:37 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices1003.wikimedia.org [production]
17:31 <andrew@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudservices1003.wikimedia.org [production]
17:30 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices1004.wikimedia.org [production]
17:24 <andrew@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudservices1004.wikimedia.org [production]
17:24 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudservices1004.wikimedia.org [production]
17:24 <andrew@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudservices1004.wikimedia.org [production]
15:57 <_joe_> uploading conftool 2.2.0 to buster, bullseye T305824 T305582 T305607 T305638 T307905 T308100 [production]
12:38 <akosiaris@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
12:38 <akosiaris@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply [production]
12:37 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
12:37 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [production]
12:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2140 after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P27824 and previous config saved to /var/cache/conftool/dbconfig/20220513-121832-marostegui.json [production]
12:09 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
11:59 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [production]
11:57 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
11:47 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [production]
11:40 <moritzm> installing idp-test1002 T308214 [production]
10:55 <moritzm> installing idp-test2002 T308214 [production]
10:41 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on ganeti4002.ulsfo.wmnet with reason: Remove from cluster for eventual reimage [production]
10:41 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on ganeti4002.ulsfo.wmnet with reason: Remove from cluster for eventual reimage [production]
10:18 <vgutierrez> disable puppet on gerrit1001 to fix /etc/ssh/ssh_config [production]
08:39 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
08:03 <jynus> moving s2 database from db2101 to db2097 T299920 [production]
07:59 <moritzm> draining ganeti4002 T307997 [production]
07:52 <XioNoX> add init7 transit in drmrs [production]
07:39 <root@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti4001.ulsfo.wmnet to ganeti01.svc.ulsfo.wmnet [production]
07:39 <root@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti4001.ulsfo.wmnet to ganeti01.svc.ulsfo.wmnet [production]
07:27 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4001.ulsfo.wmnet [production]