2851-2900 of 10000 results (50ms)
2022-03-02 §
03:48 <ladsgroup@cumin1001> START - Cookbook sre.hosts.reimage for host db1104.eqiad.wmnet with OS bullseye [production]
03:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T300992)', diff saved to https://phabricator.wikimedia.org/P21634 and previous config saved to /var/cache/conftool/dbconfig/20220302-034715-ladsgroup.json [production]
03:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1104 (T302185)', diff saved to https://phabricator.wikimedia.org/P21633 and previous config saved to /var/cache/conftool/dbconfig/20220302-034502-ladsgroup.json [production]
03:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1104.eqiad.wmnet with reason: Maintenance [production]
03:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1104.eqiad.wmnet with reason: Maintenance [production]
03:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1161 (T300992)', diff saved to https://phabricator.wikimedia.org/P21632 and previous config saved to /var/cache/conftool/dbconfig/20220302-034454-ladsgroup.json [production]
03:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
03:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
03:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
03:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
03:43 <ejegg> updated CiviCRM from e9f0eff5 to cb0605ed [production]
02:13 <ejegg> Fundraising CiviCRM updated from 2874d623 to e9f0eff5 [production]
00:15 <topranks> Re-enabling Lumen AS3356 BGP session over IPv4 on cr3-ulsfo to assess affect on currently broken routing to ulsfo. [production]
00:07 <topranks> disabling Lumen AS3356 BGP session over IPv4 on cr3-ulsfo to assess affect on currently broken routing to ulsfo. [production]
2022-03-01 §
22:50 <inflatador> T276198 reenabled puppet on elastic1052.eqiad.wmnet [production]
22:37 <inflatador> T276198 rebooting elastic1052.eqiad.wmnet to test failure condition [production]
22:33 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp6016.drmrs.wmnet with reason: debugging till we find the root cause of the purged OOM issue; no traffic served [production]
22:33 <sukhe@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cp6016.drmrs.wmnet with reason: debugging till we find the root cause of the purged OOM issue; no traffic served [production]
22:32 <inflatador> T276198 disabling puppet on elastic1052.eqiad.wmnet to test failure condition (rebooting shortly) [production]
21:53 <dancy@deploy1002> Finished scap: Resync to try to clear alerts (duration: 12m 08s) [production]
21:41 <dancy@deploy1002> Started scap: Resync to try to clear alerts [production]
21:36 <dancy@deploy1002> Started scap: Resync to try to clear alerts [production]
20:36 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.24 refs T300200 [production]
20:33 <brennen> 1.38.0-wmf.24 train (T300200): no current blockers; proceeding to group0; note this may briefly trigger some version alerts [production]
20:30 <brennen@deploy1002> Synchronized php-1.38.0-wmf.24/includes: Backport: [[gerrit:767089|Revert "preferences: Use a faster and simpler form descriptor when validating" (T302643)]] (duration: 00m 55s) [production]
20:05 <mutante> alert1001 - re-enabled puppet [production]
20:05 <brennen@deploy1002> Finished scap: testwikis wikis to 1.38.0-wmf.24 refs T300200 (duration: 53m 17s) [production]
19:45 <mutante> alert1001 - disable puppet, systemctl stop ircecho - to stop bot spam, caused somehow by new scap version breaking "mw versions mismwatch" alerting - affects labtestwiki,testwiki,testwikidatawiki [production]
19:38 <mutante> mw1449 - scap pull [production]
19:36 <mutante> mw1414 - scap pull [production]
19:11 <brennen@deploy1002> Started scap: testwikis wikis to 1.38.0-wmf.24 refs T300200 [production]
19:07 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2008.codfw.wmnet [production]
19:01 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:58 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
18:57 <brennen> 1.38.0-wmf.24 train (T300200): there's currently a single blocker at T302643; staging to testwikis and holding there until backport's available [production]
18:54 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti2008.codfw.wmnet [production]
18:45 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2008.codfw.wmnet with reason: Remove from Ganeti cluster for decom [production]
18:45 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2008.codfw.wmnet with reason: Remove from Ganeti cluster for decom [production]
18:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T300992)', diff saved to https://phabricator.wikimedia.org/P21626 and previous config saved to /var/cache/conftool/dbconfig/20220301-180216-ladsgroup.json [production]
17:52 <cwhite> completed grafana upgrade in eqiad T282863 [production]
17:50 <herron> re-enabling puppet and ircecho on alert1001 [production]
17:47 <cwhite> upgrade grafana in eqiad T282863 [production]
17:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P21625 and previous config saved to /var/cache/conftool/dbconfig/20220301-174711-ladsgroup.json [production]
17:44 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:34 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P21624 and previous config saved to /var/cache/conftool/dbconfig/20220301-173206-ladsgroup.json [production]
17:24 <dancy@deploy1002> Finished scap: testing container image build (duration: 28m 39s) [production]
17:17 <herron> stopped ircecho on alert1001 due to systemd unit alert shower [production]
17:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T300992)', diff saved to https://phabricator.wikimedia.org/P21622 and previous config saved to /var/cache/conftool/dbconfig/20220301-171701-ladsgroup.json [production]
17:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1144:3315 (T300992)', diff saved to https://phabricator.wikimedia.org/P21621 and previous config saved to /var/cache/conftool/dbconfig/20220301-171441-ladsgroup.json [production]