6451-6500 of 10000 results (95ms)
2023-02-20 §
09:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set db2161 with weight 0 T330056', diff saved to https://phabricator.wikimedia.org/P44686 and previous config saved to /var/cache/conftool/dbconfig/20230220-092727-ladsgroup.json [production]
09:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s8 T330056 [production]
09:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s8 T330056 [production]
09:09 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Move all of NS-related config out of IS.php to a dedicated file, part III (T308932) (duration: 06m 24s) [production]
09:09 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
09:06 <XioNoX> delete old group ID custom field from Netbox - https://netbox.wikimedia.org/extras/custom-fields/6/ - T260363 [production]
09:01 <ladsgroup@deploy1002> Synchronized multiversion/MWConfigCacheGenerator.php: Move all of NS-related config out of IS.php to a dedicated file, part II (T308932) (duration: 06m 47s) [production]
08:58 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
08:54 <ladsgroup@deploy1002> Synchronized wmf-config/core-Namespaces.php: Move all of NS-related config out of IS.php to a dedicated file, part I (T308932) (duration: 16m 10s) [production]
08:54 <ayounsi@cumin1001> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Update wmf-plugin - ayounsi@cumin1001 [production]
08:52 <ayounsi@cumin1001> START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Update wmf-plugin - ayounsi@cumin1001 [production]
08:52 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
08:44 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Mepps out of all services on: 946 hosts [production]
08:43 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Mepps out of all services on: 946 hosts [production]
08:42 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Mepps out of all services on: 1067 hosts [production]
08:42 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
08:41 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
08:41 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Mepps out of all services on: 1067 hosts [production]
08:41 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
08:41 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
08:41 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
08:41 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
08:41 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
08:40 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
08:40 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
08:40 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
08:40 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
08:40 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
08:40 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
08:39 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
08:39 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
08:08 <moritzm> updating openjdk-11 on elastic* servers T329957 [production]
07:44 <moritzm> imported jenkins 2.375.3 to thirdparty/ci T330045 [production]
07:41 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
07:41 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
07:40 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
07:40 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
07:40 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
07:39 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
07:39 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
07:39 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
07:27 <Amir1> running migrateTagTemplate.php on all wikis (T329766) [production]
06:40 <hashar> Restarting Gerrit [production]
2023-02-18 §
08:29 <elukey> kill leftover processes of user `mepps` (offboarded) from stat100[4,5] to unblock puppet [production]
08:24 <elukey> delete /var/log/{syslog,messages,user.log).1 on kubestagetcd1005 to free space [production]
08:22 <elukey> delete /var/log/{messages,user.log).1 on kubestageetcd1006 to free space [production]
08:21 <elukey> delete /var/log/syslog.1 on kubestageetcd1006 to free space [production]
2023-02-17 §
22:45 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T329957 [production]
22:09 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T329957 [production]
22:06 <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T329957 [production]