|
2021-06-23
ยง
|
| 22:24 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1414.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:45 |
<sbassett> |
Deployed updated security patch for T285190 to wmf.9 and wmf.11 |
[production] |
| 20:55 |
<ejegg> |
updated payments-wiki from 42cfbe832d to d9892207c1 |
[production] |
| 20:38 |
<eileen> |
civicrm revision changed from 53d103f672 to bd906975f0, config revision is 6a88618c3e |
[production] |
| 20:19 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:17 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:16 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:13 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:42 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: Revert group1 wikis to 1.37.0-wmf.9 |
[production] |
| 19:39 |
<dduvall> |
rolling back wmf.11 from group1 due to increase in logspam possibly related to noted risky patch https://gerrit.wikimedia.org/r/c/mediawiki/core/+/693298 (cc T281152 and patch contact Amir1) |
[production] |
| 19:35 |
<herron> |
rebooting kafkamon hosts for updates |
[production] |
| 19:26 |
<dduvall@deploy1002> |
Synchronized php: group1 wikis to 1.37.0-wmf.11 (duration: 01m 06s) |
[production] |
| 19:25 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.11 |
[production] |
| 19:20 |
<dduvall> |
preparing to promote wmf.11 group1 (T281152) cc'ing risky patch contacts Amir1, Krinkle, DannyS712 |
[production] |
| 19:10 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 6e0f5ad88cb7d99e3b4cf48bccb6e34cdcc64fa5: Enable GrowthExperiments donor landing page for testing (T284799) (duration: 01m 05s) |
[production] |
| 19:07 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.9/extensions/GrowthExperiments/: 2338e53: Revert "Add custom signup flow for donors" (T284740; T284800; T285281) (duration: 01m 06s) |
[production] |
| 18:58 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:55 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.9/extensions/GrowthExperiments/: REVERT: 76e5fc91083736d14049a05ed227cdea015c113e: Add custom signup flow for donors (T284740; T284800; T285281) (duration: 00m 38s) |
[production] |
| 18:55 |
<urbanecm@deploy1002> |
sync-file aborted: REVERT: 76e5fc91083736d14049a05ed227cdea015c113e: Add custom signup flow for donors (T284740; T284800; T285281) (duration: 00m 01s) |
[production] |
| 18:54 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:54 |
<urbanecm@deploy1002> |
Scap failed!: 6/9 canaries failed their endpoint checks(https://en.wikipedia.org) |
[production] |
| 18:53 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.11/extensions/GrowthExperiments/: 76e5fc91083736d14049a05ed227cdea015c113e: Add custom signup flow for donors (T284740; T284800; T285281) (duration: 01m 07s) |
[production] |
| 18:39 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.11/extensions/WikimediaEvents/extension.json: 01f034b466ff7bdd274e18c9ad7cefe88245548d: Finalize WMDEBanner* schema migration to Event Platform (T282562) (duration: 01m 05s) |
[production] |
| 18:35 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.9/extensions/GrowthExperiments/includes/Specials/SpecialEditGrowthConfig.php: 17efbafc300c6745928415d7a10f2dad8f406de4: EditGrowthConfig: Suggested edit "Learn more" link should support interwiki (T279886; T285385) (duration: 01m 06s) |
[production] |
| 18:33 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 3a2fc6e3687817db5c774e7e527a10dd9e974138: Enable $wgSecurePollSingleTransferableVoteEnabled on beta sites (duration: 01m 05s) |
[production] |
| 18:31 |
<ebernhardson@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@0535b94]: expect eventgate events for all datacenters, second try (duration: 09m 11s) |
[production] |
| 18:22 |
<ebernhardson@deploy1002> |
Started deploy [wikimedia/discovery/analytics@0535b94]: expect eventgate events for all datacenters, second try |
[production] |
| 18:09 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: b4a786799a249d2012b9c47553a0b64fdce1bac0: Make Growth features available to newcomers at lvwiki and skwiki (T278191; T284149) (duration: 01m 06s) |
[production] |
| 17:58 |
<herron> |
beginning rolling reboots of kafka-main100[1-5] for updates |
[production] |
| 17:57 |
<otto@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Enable canary events for NavigationTiming ext streams - T271208, T266798 (duration: 01m 29s) |
[production] |
| 17:07 |
<herron> |
beginning rolling reboots of kafka-main200[1-5] for updates |
[production] |
| 16:42 |
<XioNoX> |
re-start sending traffic on the codfw-eqsin Telia transport link |
[production] |
| 15:17 |
<topranks> |
Removing peering to AS64050 / "BGP Consultancy Pte Ltd" at AMS-IX (cr2-esams). Peer has left IX. |
[production] |
| 14:54 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist $SHARD recountCategories.php --mode=pages && foreachwikiindblist $SHARD recountCategories.php --mode=subcats && foreachwikiindblist $SHARD recountCategories.php --mode=files # T170737, SHARD=s1 |
[production] |
| 14:53 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist $SHARD recountCategories.php --mode=pages && foreachwikiindblist $SHARD recountCategories.php --mode=subcats && foreachwikiindblist $SHARD recountCategories.php --mode=files # T170737, SHARD=s8 |
[production] |
| 13:54 |
<effie> |
rolling restart thanos-fe* to pick up new tegola-vector-tiles account - T283049 |
[production] |
| 13:45 |
<volans> |
uploaded cumin_4.1.1 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia |
[production] |
| 13:27 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist $SHARD recountCategories.php --mode=pages && foreachwikiindblist $SHARD recountCategories.php --mode=subcats && foreachwikiindblist $SHARD recountCategories.php --mode=files # T170737, SHARD=s4 |
[production] |
| 12:59 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist $SHARD recountCategories.php --mode=pages && foreachwikiindblist $SHARD recountCategories.php --mode=subcats && foreachwikiindblist $SHARD recountCategories.php --mode=files # T170737, SHARD=s3 |
[production] |
| 12:46 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist $SHARD recountCategories.php --mode=pages && foreachwikiindblist $SHARD recountCategories.php --mode=subcats && foreachwikiindblist $SHARD recountCategories.php --mode=files # T170737, SHARD=s7 |
[production] |
| 12:35 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist $SHARD recountCategories.php --mode=pages && foreachwikiindblist $SHARD recountCategories.php --mode=subcats && foreachwikiindblist $SHARD recountCategories.php --mode=files # T170737, SHARD=s6 |
[production] |
| 12:26 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist $SHARD recountCategories.php --mode=pages && foreachwikiindblist $SHARD recountCategories.php --mode=subcats && foreachwikiindblist $SHARD recountCategories.php --mode=files # T170737, SHARD=s5 |
[production] |
| 12:15 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ foreachwikiindblist s2 recountCategories.php --mode=pages && foreachwikiindblist s2 recountCategories.php --mode=subcats && foreachwikiindblist s2 recountCategories.php --mode=files # T170737 |
[production] |
| 11:46 |
<XioNoX> |
Simplify labs-in4/6 firewall filters - CR700939 |
[production] |
| 11:10 |
<topranks> |
Removing peering to AS39651 / "Com Hem AB" at AMS-IX (cr2-esams). Peer has left IX. |
[production] |
| 10:44 |
<oblivian@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 09:35 |
<mbsantos@deploy1002> |
Finished deploy [kartotherian/deploy@9f16a6b]: (no justification provided) (duration: 00m 20s) |
[production] |
| 09:35 |
<mbsantos@deploy1002> |
Started deploy [kartotherian/deploy@9f16a6b]: (no justification provided) |
[production] |
| 09:22 |
<oblivian@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 08:48 |
<volans> |
sudo systemctl start ferm.service on thanos-fe2002 (DNS query timeout) |
[production] |