4501-4550 of 10000 results (45ms)
2022-01-12 ยง
12:51 <awight> EU deployment complete [production]
12:50 <awight@deploy1002> Synchronized php-1.38.0-wmf.17/extensions/TemplateData: Backport: [[gerrit:752775|Allow aliases to be integers in addition to strings (T298795)]] (duration: 01m 07s) [production]
12:50 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM failoid1002.eqiad.wmnet [production]
12:48 <Amir1> removing orphan lint error reports in all wikis (T298782) [production]
12:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T297191)', diff saved to https://phabricator.wikimedia.org/P18650 and previous config saved to /var/cache/conftool/dbconfig/20220112-124514-marostegui.json [production]
12:32 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:31 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P18649 and previous config saved to /var/cache/conftool/dbconfig/20220112-123010-marostegui.json [production]
12:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Give more traffic to db1128 in s1 T295965', diff saved to https://phabricator.wikimedia.org/P18648 and previous config saved to /var/cache/conftool/dbconfig/20220112-122742-marostegui.json [production]
12:15 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P18647 and previous config saved to /var/cache/conftool/dbconfig/20220112-121505-marostegui.json [production]
12:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:13 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: cfe389afce8037121f8e8b672f4fdf2458a068dd: fawiki: Add extendedmover usergroup (T299038) (duration: 01m 08s) [production]
12:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:10 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM doc1002.eqiad.wmnet [production]
12:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Give more traffic to db1128 in s1 T295965', diff saved to https://phabricator.wikimedia.org/P18646 and previous config saved to /var/cache/conftool/dbconfig/20220112-120931-marostegui.json [production]
12:08 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM doc1002.eqiad.wmnet [production]
12:06 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM doc1001.eqiad.wmnet [production]
12:03 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM doc1001.eqiad.wmnet [production]
12:00 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM releases1002.eqiad.wmnet [production]
12:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T297191)', diff saved to https://phabricator.wikimedia.org/P18645 and previous config saved to /var/cache/conftool/dbconfig/20220112-120000-marostegui.json [production]
11:58 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM releases1002.eqiad.wmnet [production]
11:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Give more traffic to db1128 in s1 T295965', diff saved to https://phabricator.wikimedia.org/P18644 and previous config saved to /var/cache/conftool/dbconfig/20220112-115259-marostegui.json [production]
11:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1175 (T297191)', diff saved to https://phabricator.wikimedia.org/P18643 and previous config saved to /var/cache/conftool/dbconfig/20220112-115031-marostegui.json [production]
11:50 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
11:50 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
11:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179 (T297191)', diff saved to https://phabricator.wikimedia.org/P18642 and previous config saved to /var/cache/conftool/dbconfig/20220112-115024-marostegui.json [production]
11:42 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/shellbox-media: sync on main [production]
11:42 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/shellbox-media: apply on main [production]
11:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P18641 and previous config saved to /var/cache/conftool/dbconfig/20220112-113518-marostegui.json [production]
11:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Give more traffic to db1128 in s1 T295965', diff saved to https://phabricator.wikimedia.org/P18640 and previous config saved to /var/cache/conftool/dbconfig/20220112-113119-marostegui.json [production]
11:21 <elukey> move kafka-jumbo nodes to fixed kafka uid/gid - T296990 [production]
11:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P18639 and previous config saved to /var/cache/conftool/dbconfig/20220112-112013-marostegui.json [production]
11:07 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
11:06 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
11:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
11:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179 (T297191)', diff saved to https://phabricator.wikimedia.org/P18638 and previous config saved to /var/cache/conftool/dbconfig/20220112-110508-marostegui.json [production]
11:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
11:04 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM dborch1001.wikimedia.org [production]
11:02 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM dborch1001.wikimedia.org [production]
10:59 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
10:59 <moritzm> rebalance ganeti/codfw row B (all nodes reimaged to Buster) [production]
10:58 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
10:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
10:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Give more traffic to db1128 in s1 T295965', diff saved to https://phabricator.wikimedia.org/P18637 and previous config saved to /var/cache/conftool/dbconfig/20220112-105650-marostegui.json [production]
10:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1179 (T297191)', diff saved to https://phabricator.wikimedia.org/P18636 and previous config saved to /var/cache/conftool/dbconfig/20220112-105540-marostegui.json [production]
10:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance [production]