51-100 of 10000 results (25ms)
2021-11-11 §
12:21 <moritzm> imported openjdk-8 8u312-b07-1~deb10u1 to component/jdk8 for buster-wikimedia (rebuild of latest Java 8 security release for Buster) [production]
12:17 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:15 <awight@deploy1002> Synchronized multiversion/buildConfigCache.php: Config: [[gerrit:737189|Don't need to keep all config in memory]] (duration: 00m 55s) [production]
12:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:13 <awight@deploy1002> Synchronized multiversion/MWConfigCacheGenerator.php: Config: [[gerrit:737192|Avoid error suppression]] (duration: 00m 55s) [production]
12:10 <jynus@cumin2002> START - Cookbook sre.hosts.reimage for host db2097.codfw.wmnet with OS buster [production]
12:10 <jynus@cumin1001> START - Cookbook sre.hosts.reimage for host db1116.eqiad.wmnet with OS buster [production]
12:08 <awight@deploy1002> Synchronized multiversion/buildConfigCache.php: Config: [[gerrit:737187|Anchor relative import]] (duration: 00m 56s) [production]
11:32 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests [production]
11:31 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests [production]
11:28 <jynus@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1001.eqiad.wmnet with OS buster [production]
11:04 <jynus@cumin1001> START - Cookbook sre.hosts.reimage for host dbprov1001.eqiad.wmnet with OS buster [production]
10:56 <jynus@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2001.codfw.wmnet with OS buster [production]
10:37 <moritzm> updated routinator in thirdparty/routinator for bullseye-wikimedia to 0.10.12 T292503 [production]
10:24 <jynus@cumin2002> START - Cookbook sre.hosts.reimage for host dbprov2001.codfw.wmnet with OS buster [production]
10:18 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3065.esams.wmnet with OS buster [production]
10:15 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests [production]
10:15 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests [production]
10:15 <vgutierrez> pool cp3065 running haproxy - T290005 [production]
09:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove contributions from s5 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17725 and previous config saved to /var/cache/conftool/dbconfig/20211111-092528-marostegui.json [production]
09:13 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reimage for host cp3065.esams.wmnet with OS buster [production]
09:10 <vgutierrez> depool cp3065 to be reimaged as cache::upload_haproxy - T290005 [production]
09:03 <arturo> pull all packages for buster-wikimedia/thirdparty/kubeadm-k8s-1-21 (T282942) [production]
08:17 <marostegui> Upgrade db2078 T288720 [production]
08:13 <marostegui> Restart db1132 T288720 [production]
06:56 <elukey> `systemctl start prometheus-mysqld-exporter@analytics_meta` on db1108 [production]
06:37 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1104.eqiad.wmnet with OS buster [production]
06:10 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1104.eqiad.wmnet with OS buster [production]
06:06 <marostegui> Stop replication on db1104 (old master) T294321 [production]
06:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1104 (old master) T294321', diff saved to https://phabricator.wikimedia.org/P17723 and previous config saved to /var/cache/conftool/dbconfig/20211111-060242-marostegui.json [production]
06:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1109 to s8 primary and set section read-write T294321', diff saved to https://phabricator.wikimedia.org/P17722 and previous config saved to /var/cache/conftool/dbconfig/20211111-060102-marostegui.json [production]
06:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - T294321', diff saved to https://phabricator.wikimedia.org/P17721 and previous config saved to /var/cache/conftool/dbconfig/20211111-060031-marostegui.json [production]
06:00 <marostegui> Starting s8 eqiad failover from db1104 to db1109 - T294321 [production]
05:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321 [production]
05:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321 [production]
02:52 <eileen> civicrm revision 7e38867f -> 817e514a (latest) [production]
00:22 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:18 <reedy@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Set wgForeignUploadTargets on officewiki T295510 (duration: 00m 56s) [production]
2021-11-10 §
23:46 <ebernhardson> start test backup/restore of 1tb commonswiki from relforge to swift in eqiad [production]
23:33 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript updateSpecialPages.php --wiki=foundationwiki --only=DoubleRedirects [production]
23:33 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript updateSpecialPages.php --wiki=foundationwiki --only=BrokenRedirects [production]
22:06 <bblack> dns2002 - restart ntp.servce to fix drmrs peering [production]
22:01 <bblack> dns1002 - restart ntp.servce to fix drmrs peering [production]
21:56 <bblack> dns2001 - restart ntp.service to fix drmrs peering [production]
21:53 <bblack> dns1001 - restart ntp.service to see if drmrs associations cleared up after dns changes, etc [production]
21:24 <bblack> asw1-b1[23]-drmrs: added ipv6 router-advertisement clauses, which work, but probably imperfectly :) [production]
19:52 <bblack@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6001.wikimedia.org with OS buster [production]
19:51 <bblack@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6002.wikimedia.org with OS buster [production]
19:51 <ottomata> altering {eqiad,codfw}.maps.tiles_change to increase to 6 partitions in kafka main-eqiad, main-codfw and jumbo-eqiad: https://phabricator.wikimedia.org/T293366#7497076 [production]