8801-8850 of 10000 results (41ms)
2021-05-11 §
12:11 <jmm@cumin2002> END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mc1027.eqiad.wmnet [production]
12:11 <jmm@cumin2002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mc1027.eqiad.wmnet [production]
11:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 100%: Repool db1162', diff saved to https://phabricator.wikimedia.org/P15913 and previous config saved to /var/cache/conftool/dbconfig/20210511-114540-root.json [production]
11:35 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw1002.eqiad.wmnet [production]
11:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 75%: Repool db1162', diff saved to https://phabricator.wikimedia.org/P15912 and previous config saved to /var/cache/conftool/dbconfig/20210511-113036-root.json [production]
11:16 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:688178|Add P2671 and P4839 to deprecated properties list (T280779)]] (duration: 00m 58s) [production]
11:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 50%: Repool db1162', diff saved to https://phabricator.wikimedia.org/P15911 and previous config saved to /var/cache/conftool/dbconfig/20210511-111532-root.json [production]
11:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 25%: Repool db1162', diff saved to https://phabricator.wikimedia.org/P15910 and previous config saved to /var/cache/conftool/dbconfig/20210511-110029-root.json [production]
10:52 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
10:46 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
10:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1162', diff saved to https://phabricator.wikimedia.org/P15909 and previous config saved to /var/cache/conftool/dbconfig/20210511-102303-marostegui.json [production]
10:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 100%: Repool db1170:3312', diff saved to https://phabricator.wikimedia.org/P15908 and previous config saved to /var/cache/conftool/dbconfig/20210511-102212-root.json [production]
10:13 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudgw1002.eqiad.wmnet [production]
10:13 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
10:07 <jayme@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
10:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 75%: Repool db1170:3312', diff saved to https://phabricator.wikimedia.org/P15907 and previous config saved to /var/cache/conftool/dbconfig/20210511-100708-root.json [production]
09:54 <aborrero@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cloudgw2002-dev.codfw.wmnet [production]
09:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 50%: Repool db1170:3312', diff saved to https://phabricator.wikimedia.org/P15904 and previous config saved to /var/cache/conftool/dbconfig/20210511-095204-root.json [production]
09:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 25%: Repool db1170:3312', diff saved to https://phabricator.wikimedia.org/P15903 and previous config saved to /var/cache/conftool/dbconfig/20210511-093701-root.json [production]
09:23 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudgw2002-dev.codfw.wmnet [production]
08:37 <jayme@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:36 <jayme@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
08:35 <jayme@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
08:34 <jayme@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
08:32 <moritzm> installing hivex security updates [production]
08:31 <jayme@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [production]
08:30 <jayme@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [production]
08:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1170:3312', diff saved to https://phabricator.wikimedia.org/P15901 and previous config saved to /var/cache/conftool/dbconfig/20210511-082038-marostegui.json [production]
08:19 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
08:17 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
07:55 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
07:54 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
07:40 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
07:39 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
07:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1182 (re)pooling @ 100%: Repool db1182', diff saved to https://phabricator.wikimedia.org/P15899 and previous config saved to /var/cache/conftool/dbconfig/20210511-070742-root.json [production]
06:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1182 (re)pooling @ 75%: Repool db1182', diff saved to https://phabricator.wikimedia.org/P15898 and previous config saved to /var/cache/conftool/dbconfig/20210511-065238-root.json [production]
06:50 <marostegui> Stop replication on db2094:3318 T282514 [production]
06:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1182 (re)pooling @ 50%: Repool db1182', diff saved to https://phabricator.wikimedia.org/P15897 and previous config saved to /var/cache/conftool/dbconfig/20210511-063734-root.json [production]
06:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1182 (re)pooling @ 25%: Repool db1182', diff saved to https://phabricator.wikimedia.org/P15896 and previous config saved to /var/cache/conftool/dbconfig/20210511-062231-root.json [production]
05:44 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1082.eqiad.wmnet [production]
05:36 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db1082.eqiad.wmnet [production]
05:32 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1121.eqiad.wmnet with reason: REIMAGE [production]
05:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1121.eqiad.wmnet with reason: REIMAGE [production]
05:11 <marostegui> Reimage db1121 to buster, this will generate lag on s4 (commonswiki) on wikireplicas T280492 [production]
05:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121 - going to be reimaged to buster T280492', diff saved to https://phabricator.wikimedia.org/P15895 and previous config saved to /var/cache/conftool/dbconfig/20210511-051102-marostegui.json [production]
05:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1182', diff saved to https://phabricator.wikimedia.org/P15894 and previous config saved to /var/cache/conftool/dbconfig/20210511-050816-marostegui.json [production]
2021-05-10 §
23:38 <urbanecm@deploy1002> Synchronized wmf-config/CommonSettings.php: 779fb53bfd7a4d9b11f865df14f8a72adb97f33b: Update messages used for tech CoC (T280886) (duration: 00m 56s) [production]
23:32 <urbanecm@deploy1002> Synchronized wmf-config/extension-list: ba8b786c7f3a290f0747a6859fd07502eb83108f: NO-OP: Enable ChessBrowser on beta (T244075) (duration: 00m 57s) [production]
23:12 <urbanecm@deploy1002> Synchronized wmf-config/logos.php: dd6fa6504350a90c9f14c218bc972558791f0a6d: Use ptwiki 20th anniversary logos (T281925) (duration: 00m 59s) [production]
23:08 <urbanecm@deploy1002> Synchronized static/images/project-logos/: f2a76b1a6eb55749395e67d74c74a7fc5df52f1b: Add ptwiki 20th anniversary logos (T281925) (duration: 00m 58s) [production]