3451-3500 of 10000 results (64ms)
2019-09-24 §
08:32 <mvolz@deploy1001> scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad] [production]
08:30 <marostegui> Deploy schema change on s4 master with replication - T231172 [production]
08:29 <effie> Disable puppet on api cluster and restart php-fpm to finish php7 migration - T219150 [production]
08:19 <mvolz@deploy1001> scap-helm zotero finished [production]
08:19 <mvolz@deploy1001> scap-helm zotero cluster staging completed [production]
08:19 <mvolz@deploy1001> scap-helm zotero upgrade staging -f zotero-values-staging.yaml stable/zotero [namespace: zotero, clusters: staging] [production]
08:18 <marostegui> Deploy schema change on s5 master with replication - T231172 [production]
07:51 <onimisionipe> depool wdqs1006 to clear HTTP too many request error [production]
07:43 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) [production]
07:42 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:38 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:38 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:33 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) [production]
07:32 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:29 <moritzm> uploaded openjdk-8 8u222-b10-1~deb10u2 to buster-wikimedia component/jdk8 T233604 [production]
07:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:24 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:18 <godog> swift eqiad-prod: continue ms-be1027 decom T233289 [production]
06:40 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) [production]
06:40 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:37 <marostegui> Stop MySQL on db1066 - T233071 [production]
06:36 <marostegui> Remove db1066 from tendril and zarcillo T233071 [production]
06:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1075', diff saved to https://phabricator.wikimedia.org/P9163 and previous config saved to /var/cache/conftool/dbconfig/20190924-063002-marostegui.json [production]
06:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Give more weight to db1075', diff saved to https://phabricator.wikimedia.org/P9162 and previous config saved to /var/cache/conftool/dbconfig/20190924-061943-marostegui.json [production]
05:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Give more weight to db1075', diff saved to https://phabricator.wikimedia.org/P9161 and previous config saved to /var/cache/conftool/dbconfig/20190924-053919-marostegui.json [production]
05:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Give weight 100 to db1075', diff saved to https://phabricator.wikimedia.org/P9160 and previous config saved to /var/cache/conftool/dbconfig/20190924-052545-marostegui.json [production]
05:13 <cdanis@cumin1001> dbctl commit (dc=all): 're-do T230783 master promotion and set read-write', diff saved to https://phabricator.wikimedia.org/P9159 and previous config saved to /var/cache/conftool/dbconfig/20190924-051307-cdanis.json [production]
05:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1123 to s3 master and remove read-only from s3 T230783', diff saved to https://phabricator.wikimedia.org/P9158 and previous config saved to /var/cache/conftool/dbconfig/20190924-051147-marostegui.json [production]
05:10 <cdanis> T230783 mark DEFAULT not s3 as readonly in etcd etcd dbconfig data [production]
05:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s3 as read-only for maintenance T230783', diff saved to https://phabricator.wikimedia.org/P9157 and previous config saved to /var/cache/conftool/dbconfig/20190924-050034-marostegui.json [production]
05:00 <marostegui> Starting s3 failover from db1075 to db1123 - T230783 [production]
04:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Set weight 0 to db1123 T230783', diff saved to https://phabricator.wikimedia.org/P9156 and previous config saved to /var/cache/conftool/dbconfig/20190924-042121-marostegui.json [production]
04:13 <marostegui> Start pre switchover steps - T230783 [production]
03:52 <chaomodus> rebooted netboxdb[12]001 for kernel upgrade [production]
03:46 <crusnov@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
03:45 <crusnov@cumin1001> START - Cookbook sre.hosts.downtime [production]
00:43 <mutante> db2060 - remove PXE flag boot override - set Boot Device to none [production]
2019-09-23 §
23:56 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
23:50 <dzahn@cumin1001> Updating IPMI password on 92 hosts - dzahn@cumin1001 [production]
23:50 <dzahn@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
23:46 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
23:43 <dzahn@cumin1001> Updating IPMI password on 92 hosts - dzahn@cumin1001 [production]
23:43 <dzahn@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
21:32 <catrope@deploy1001> Synchronized wmf-config/VariantSettings.php: Syncing no-op change for T232419 (duration: 00m 57s) [production]
19:57 <cdanis> T233657 ✔️ cdanis@cp4027.ulsfo.wmnet ~ 🕓🍵 sudo -i depool [production]
19:16 <urbanecm@deploy1001> Synchronized wmf-config/VariantSettings.php: 2a7a125: Redefine hiwikisource extra namespaces (T233365) (duration: 00m 57s) [production]
19:09 <Urbanecm> Going to deploy one more last-time patch [production]
18:51 <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: MachineVision: Update active handler config, take 2 (T233610) (duration: 00m 56s) [production]
18:48 <Urbanecm> Morning SWAT done [production]
18:48 <urbanecm@deploy1001> Synchronized wmf-config/VariantSettings.php: SWAT: 37fcbdf: Fix: Move hiwikisource extra namespace to extra namespace section (duration: 00m 56s) [production]