501-550 of 10000 results (65ms)
2022-10-20 §
06:53 <kart_> Updated Updated cxserver to 2022-10-18-161640-production (T317224, T319175, T319176) [production]
06:52 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
06:51 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
06:48 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
06:47 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
06:42 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
06:41 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
2022-10-19 §
23:33 <wfan> civicrm upgraded from 477323fe to c96dd3ae [production]
23:24 <ejegg> re-enabled refund queue consumer job [production]
22:43 <ejegg> updated fundraising CiviCRM from 4b9e981a to 477323fe [production]
22:33 <ejegg> disabled fundraising refund QC, started language update job [production]
22:02 <ejegg> updated standalone SmashPig IPN listener from f36143f0 to 9295dc2a [production]
22:00 <jhathaway@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-etcd1003.eqiad.wmnet [production]
21:49 <eileen> config revision changed from 903c8ce2 to 4aad20b1 [production]
21:37 <jhathaway@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-etcd1003.eqiad.wmnet on all recursors [production]
21:36 <jhathaway@cumin1001> START - Cookbook sre.dns.wipe-cache aux-k8s-etcd1003.eqiad.wmnet on all recursors [production]
21:36 <jhathaway@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:30 <jhathaway@cumin1001> START - Cookbook sre.dns.netbox [production]
21:30 <jhathaway@cumin1001> START - Cookbook sre.ganeti.makevm for new host aux-k8s-etcd1003.eqiad.wmnet [production]
21:30 <jhathaway@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-etcd1002.eqiad.wmnet [production]
21:06 <jhathaway@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-etcd1002.eqiad.wmnet on all recursors [production]
21:06 <jhathaway@cumin1001> START - Cookbook sre.dns.wipe-cache aux-k8s-etcd1002.eqiad.wmnet on all recursors [production]
21:06 <jhathaway@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:03 <jhathaway@cumin1001> START - Cookbook sre.dns.netbox [production]
21:03 <jhathaway@cumin1001> START - Cookbook sre.ganeti.makevm for new host aux-k8s-etcd1002.eqiad.wmnet [production]
20:49 <mutante> lvs2010, lvs2008 - systemctl restart pybal.service ; ipvsadm -Dt '208.80.153.250:22' ; ipvsadm -Dt '[2620:0:860:ed1a::3:fa]:22' - T296022 [production]
20:44 <mutante> lvs1020, lvs1018 - systemctl restart pybal.service ; ipvsadm -Dt '208.80.154.250:22' ; ipvsadm -Dt '[2620:0:861:ed1a::3:16]:22' - T296022 [production]
20:38 <mutante> puppetmaster1001/puppetmaster2001 - delete .git-*.err files in /var/run/confd-template T296022 [production]
20:33 <TheresNoTime> closing UTC late backport window [production]
20:31 <ejegg> updated payments-wiki from 4e1f308b to 9fa4abd7 [production]
20:27 <mutante> lvs2010 - restarted pybal, removed git-ssh IP with ipvsadm [production]
20:08 <samtar@deploy1002> samtar and samtar: Backport for [[gerrit:844040|Hooks: Log to statsd when a page is noindex'd (T310974)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
20:07 <samtar@deploy1002> Started scap: Backport for [[gerrit:844040|Hooks: Log to statsd when a page is noindex'd (T310974)]] [production]
19:46 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
19:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
19:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200 (T318955)', diff saved to https://phabricator.wikimedia.org/P35617 and previous config saved to /var/cache/conftool/dbconfig/20221019-194546-ladsgroup.json [production]
19:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P35616 and previous config saved to /var/cache/conftool/dbconfig/20221019-193039-ladsgroup.json [production]
19:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P35615 and previous config saved to /var/cache/conftool/dbconfig/20221019-191533-ladsgroup.json [production]
19:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200 (T318955)', diff saved to https://phabricator.wikimedia.org/P35614 and previous config saved to /var/cache/conftool/dbconfig/20221019-190026-ladsgroup.json [production]
18:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1200 (T318955)', diff saved to https://phabricator.wikimedia.org/P35613 and previous config saved to /var/cache/conftool/dbconfig/20221019-185813-ladsgroup.json [production]
18:58 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
18:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
18:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318955)', diff saved to https://phabricator.wikimedia.org/P35612 and previous config saved to /var/cache/conftool/dbconfig/20221019-185752-ladsgroup.json [production]
18:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P35611 and previous config saved to /var/cache/conftool/dbconfig/20221019-184245-ladsgroup.json [production]
18:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P35610 and previous config saved to /var/cache/conftool/dbconfig/20221019-182739-ladsgroup.json [production]
18:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318955)', diff saved to https://phabricator.wikimedia.org/P35609 and previous config saved to /var/cache/conftool/dbconfig/20221019-181232-ladsgroup.json [production]
18:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1185 (T318955)', diff saved to https://phabricator.wikimedia.org/P35608 and previous config saved to /var/cache/conftool/dbconfig/20221019-181019-ladsgroup.json [production]
18:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance [production]
18:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance [production]
18:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318955)', diff saved to https://phabricator.wikimedia.org/P35607 and previous config saved to /var/cache/conftool/dbconfig/20221019-180958-ladsgroup.json [production]