501-550 of 10000 results (72ms)
2024-04-03 ยง
19:02 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/zotero: apply [production]
19:01 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/zotero: apply [production]
18:58 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P59383 and previous config saved to /var/cache/conftool/dbconfig/20240403-185821-arnaudb.json [production]
18:57 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
18:57 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
18:51 <eileen> config revision changed from 821f145b to 3ed18c47 [production]
18:49 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
18:49 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
18:43 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P59382 and previous config saved to /var/cache/conftool/dbconfig/20240403-184313-arnaudb.json [production]
18:37 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts restbase[1019-1027].eqiad.wmnet [production]
18:37 <eevans@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:37 <eevans@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: restbase[1019-1027].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1002" [production]
18:35 <eevans@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: restbase[1019-1027].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1002" [production]
18:34 <eevans@cumin1002> START - Cookbook sre.dns.netbox [production]
18:28 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59381 and previous config saved to /var/cache/conftool/dbconfig/20240403-182806-arnaudb.json [production]
18:25 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59380 and previous config saved to /var/cache/conftool/dbconfig/20240403-182543-arnaudb.json [production]
18:25 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance [production]
18:25 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance [production]
18:25 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59379 and previous config saved to /var/cache/conftool/dbconfig/20240403-182520-arnaudb.json [production]
18:13 <logmsgbot> dreamyjazz Deployed security patch for T361479 [production]
18:10 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P59378 and previous config saved to /var/cache/conftool/dbconfig/20240403-181013-arnaudb.json [production]
18:09 <mvolz@deploy1002> helmfile [eqiad] DONE helmfile.d/services/citoid: apply [production]
18:08 <mvolz@deploy1002> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
18:07 <mvolz@deploy1002> helmfile [codfw] DONE helmfile.d/services/citoid: apply [production]
18:07 <eevans@cumin1002> START - Cookbook sre.hosts.decommission for hosts restbase[1019-1027].eqiad.wmnet [production]
18:06 <mvolz@deploy1002> helmfile [codfw] START helmfile.d/services/citoid: apply [production]
18:05 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
18:04 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
18:03 <marostegui@cumin1002> dbctl commit (dc=all): 'db1167 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59377 and previous config saved to /var/cache/conftool/dbconfig/20240403-180323-root.json [production]
17:55 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P59375 and previous config saved to /var/cache/conftool/dbconfig/20240403-175505-arnaudb.json [production]
17:48 <marostegui@cumin1002> dbctl commit (dc=all): 'db1167 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59374 and previous config saved to /var/cache/conftool/dbconfig/20240403-174817-root.json [production]
17:39 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59373 and previous config saved to /var/cache/conftool/dbconfig/20240403-173958-arnaudb.json [production]
17:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59372 and previous config saved to /var/cache/conftool/dbconfig/20240403-173835-arnaudb.json [production]
17:38 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
17:38 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
17:38 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
17:38 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
17:37 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59371 and previous config saved to /var/cache/conftool/dbconfig/20240403-173756-arnaudb.json [production]
17:33 <marostegui@cumin1002> dbctl commit (dc=all): 'db1167 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59370 and previous config saved to /var/cache/conftool/dbconfig/20240403-173312-root.json [production]
17:22 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P59369 and previous config saved to /var/cache/conftool/dbconfig/20240403-172249-arnaudb.json [production]
17:18 <marostegui@cumin1002> dbctl commit (dc=all): 'db1167 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59368 and previous config saved to /var/cache/conftool/dbconfig/20240403-171806-root.json [production]
17:07 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P59367 and previous config saved to /var/cache/conftool/dbconfig/20240403-170741-arnaudb.json [production]
17:04 <herron> performing rolling memory upgrades on prometheus100[56] T360687 [production]
17:03 <marostegui@cumin1002> dbctl commit (dc=all): 'db1167 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59366 and previous config saved to /var/cache/conftool/dbconfig/20240403-170300-root.json [production]
16:59 <pfischer@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:59 <pfischer@deploy1002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:54 <pfischer@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:54 <pfischer@deploy1002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:52 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59365 and previous config saved to /var/cache/conftool/dbconfig/20240403-165234-arnaudb.json [production]
16:50 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59364 and previous config saved to /var/cache/conftool/dbconfig/20240403-165011-arnaudb.json [production]