1051-1100 of 10000 results (82ms)
2024-04-16 ยง
17:17 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
17:17 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
17:17 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197 (T361627)', diff saved to https://phabricator.wikimedia.org/P60675 and previous config saved to /var/cache/conftool/dbconfig/20240416-171738-marostegui.json [production]
17:16 <btullis@cumin1002> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. [production]
17:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1196 (T352010)', diff saved to https://phabricator.wikimedia.org/P60674 and previous config saved to /var/cache/conftool/dbconfig/20240416-171047-ladsgroup.json [production]
17:10 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
17:10 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
17:10 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
17:10 <marostegui@cumin1002> dbctl commit (dc=all): 'db2127 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60673 and previous config saved to /var/cache/conftool/dbconfig/20240416-171010-root.json [production]
17:10 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
17:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T352010)', diff saved to https://phabricator.wikimedia.org/P60672 and previous config saved to /var/cache/conftool/dbconfig/20240416-171006-ladsgroup.json [production]
17:04 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host cloudcontrol2009-dev.mgmt.codfw.wmnet with reboot policy FORCED [production]
17:02 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:02 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcontrol2009 DNS add - pt1979@cumin2002" [production]
17:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60671 and previous config saved to /var/cache/conftool/dbconfig/20240416-170231-marostegui.json [production]
17:01 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcontrol2009 DNS add - pt1979@cumin2002" [production]
16:59 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
16:55 <marostegui@cumin1002> dbctl commit (dc=all): 'db2127 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60670 and previous config saved to /var/cache/conftool/dbconfig/20240416-165504-root.json [production]
16:54 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60669 and previous config saved to /var/cache/conftool/dbconfig/20240416-165458-ladsgroup.json [production]
16:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60668 and previous config saved to /var/cache/conftool/dbconfig/20240416-164722-marostegui.json [production]
16:39 <marostegui@cumin1002> dbctl commit (dc=all): 'db2127 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60666 and previous config saved to /var/cache/conftool/dbconfig/20240416-163958-root.json [production]
16:39 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60665 and previous config saved to /var/cache/conftool/dbconfig/20240416-163951-ladsgroup.json [production]
16:32 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197 (T361627)', diff saved to https://phabricator.wikimedia.org/P60664 and previous config saved to /var/cache/conftool/dbconfig/20240416-163215-marostegui.json [production]
16:29 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2123 (re)pooling @ 100%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60663 and previous config saved to /var/cache/conftool/dbconfig/20240416-162926-arnaudb.json [production]
16:29 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1197 (T361627)', diff saved to https://phabricator.wikimedia.org/P60662 and previous config saved to /var/cache/conftool/dbconfig/20240416-162900-marostegui.json [production]
16:28 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance [production]
16:28 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance [production]
16:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1188 (T361627)', diff saved to https://phabricator.wikimedia.org/P60661 and previous config saved to /var/cache/conftool/dbconfig/20240416-162838-marostegui.json [production]
16:24 <marostegui@cumin1002> dbctl commit (dc=all): 'db2127 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60660 and previous config saved to /var/cache/conftool/dbconfig/20240416-162452-root.json [production]
16:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T352010)', diff saved to https://phabricator.wikimedia.org/P60659 and previous config saved to /var/cache/conftool/dbconfig/20240416-162443-ladsgroup.json [production]
16:16 <brennen> finished phabricator deploy for T362689 - believe things are currently stable [production]
16:14 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2123 (re)pooling @ 75%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60657 and previous config saved to /var/cache/conftool/dbconfig/20240416-161420-arnaudb.json [production]
16:14 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
16:13 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
16:13 <brennen@deploy1002> Finished deploy [phabricator/deployment@098b9c2]: deploy phab1004 for T362689 (duration: 00m 42s) [production]
16:13 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60656 and previous config saved to /var/cache/conftool/dbconfig/20240416-161330-marostegui.json [production]
16:13 <brennen@deploy1002> Started deploy [phabricator/deployment@098b9c2]: deploy phab1004 for T362689 [production]
16:13 <rzl> rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_globalblocking-fixGlobalBlockWhitelist.service # T360516 [production]
16:12 <brennen@deploy1002> Finished deploy [phabricator/deployment@098b9c2]: test deploy phab2002 for T362689 (duration: 00m 32s) [production]
16:12 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2217.codfw.wmnet [production]
16:12 <brennen@deploy1002> Started deploy [phabricator/deployment@098b9c2]: test deploy phab2002 for T362689 [production]
16:09 <marostegui@cumin1002> dbctl commit (dc=all): 'db2127 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60655 and previous config saved to /var/cache/conftool/dbconfig/20240416-160946-root.json [production]
16:07 <brennen> starting phabricator deploy for T362689 [production]
16:06 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
16:05 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
16:01 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues [production]
16:00 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues [production]
15:59 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2123 (re)pooling @ 50%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60654 and previous config saved to /var/cache/conftool/dbconfig/20240416-155914-arnaudb.json [production]
15:58 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host db2217.codfw.wmnet [production]
15:58 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60653 and previous config saved to /var/cache/conftool/dbconfig/20240416-155823-marostegui.json [production]