4601-4650 of 10000 results (70ms)
2022-06-15 ยง
15:34 <jynus> stopping replication for m3 on db1117, db2078 [production]
15:29 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet [production]
15:24 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet [production]
15:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128 (T310011)', diff saved to https://phabricator.wikimedia.org/P29841 and previous config saved to /var/cache/conftool/dbconfig/20220615-152315-marostegui.json [production]
15:20 <mvernon@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host ms-be1059.eqiad.wmnet with OS bullseye [production]
15:20 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phabricator.wikimedia.org with reason: maintenace [production]
15:20 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on phabricator.wikimedia.org with reason: maintenace [production]
15:06 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventstreams: apply [production]
15:05 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab1001.eqiad.wmnet with reason: maintenance [production]
15:05 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on phab1001.eqiad.wmnet with reason: maintenance [production]
15:03 <mutante> phabricator maintenance about to start [production]
15:00 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet [production]
15:00 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1059.eqiad.wmnet with reason: host reimage [production]
14:59 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
14:59 <jbond@cumin1001> Updating IPMI password on 1 hosts - jbond@cumin1001 [production]
14:58 <jbond@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
14:58 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
14:57 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1059.eqiad.wmnet with reason: host reimage [production]
14:55 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet [production]
14:54 <jbond@cumin1001> END (PASS) - Cookbook sre.pdus.rotate-password (exit_code=0) [production]
14:53 <jbond@cumin1001> START - Cookbook sre.pdus.rotate-password [production]
14:53 <jbond@cumin1001> END (PASS) - Cookbook sre.pdus.rotate-password (exit_code=0) [production]
14:53 <jbond@cumin1001> START - Cookbook sre.pdus.rotate-password [production]
14:53 <jbond@cumin1001> END (FAIL) - Cookbook sre.pdus.rotate-password (exit_code=99) [production]
14:53 <jbond@cumin1001> START - Cookbook sre.pdus.rotate-password [production]
14:52 <jbond@cumin1001> END (ERROR) - Cookbook sre.pdus.uptime (exit_code=97) [production]
14:51 <jbond@cumin1001> START - Cookbook sre.pdus.uptime [production]
14:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1128 (T310011)', diff saved to https://phabricator.wikimedia.org/P29840 and previous config saved to /var/cache/conftool/dbconfig/20220615-145028-marostegui.json [production]
14:50 <urandom> ALTER-ing replication for codfw (Cassandra) expansion -- T307641 [production]
14:50 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1128.eqiad.wmnet with reason: Maintenance [production]
14:50 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1128.eqiad.wmnet with reason: Maintenance [production]
14:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134 (T310011)', diff saved to https://phabricator.wikimedia.org/P29839 and previous config saved to /var/cache/conftool/dbconfig/20220615-145020-marostegui.json [production]
14:49 <jbond@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync data - jbond@cumin1001" [production]
14:49 <jbond@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync data - jbond@cumin1001" [production]
14:47 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
14:46 <jbond@cumin1001> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "sync data - jbond@cumin1001" [production]
14:46 <jbond@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync data - jbond@cumin1001" [production]
14:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P29838 and previous config saved to /var/cache/conftool/dbconfig/20220615-143515-marostegui.json [production]
14:34 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:31 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:30 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage [production]
14:30 <hnowlan@deploy1002> Synchronized private/PrivateSettings.php: T308670 credentials to access the similar-users service (duration: 03m 32s) [production]
14:27 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage [production]
14:23 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:22 <jbond@cumin1001> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "sync data - jbond@cumin1001" [production]
14:21 <jbond@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync data - jbond@cumin1001" [production]
14:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P29836 and previous config saved to /var/cache/conftool/dbconfig/20220615-142010-marostegui.json [production]
14:19 <jbond@cumin1001> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "sync data - jbond@cumin1001" [production]
14:19 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:18 <jbond@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync data - jbond@cumin1001" [production]