|
2020-04-14
§
|
| 15:11 |
<hnowlan@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 14:44 |
<ppchelko@deploy1001> |
Finished deploy [changeprop/deploy@354ae2d]: Remove rules enabled in k8s T248677 (duration: 01m 58s) |
[production] |
| 14:42 |
<ppchelko@deploy1001> |
Started deploy [changeprop/deploy@354ae2d]: Remove rules enabled in k8s T248677 |
[production] |
| 14:34 |
<godog> |
power down ms-be1023 - T249174 |
[production] |
| 14:33 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:33 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:33 |
<filippo@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 14:33 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:15 |
<elukey> |
enable TLS between weblog1001,mwlog2001.codfw.wmnet,mwlog1001 and Kafka Jumbo/Logging - T250147 |
[production] |
| 14:15 |
<hashar> |
Rebasing mediawiki-config on deploy1001 for a deployment-prep config change ( https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/588706/ ) |
[production] |
| 14:12 |
<ema> |
cp3050: resume purged testing T249583 |
[production] |
| 13:55 |
<ema> |
upload purged 0.4 to buster-wikimedia T249583 |
[production] |
| 13:21 |
<hashar> |
Starting zuul-merger on contint2001 |
[production] |
| 12:50 |
<vgutierrez> |
Enable inbound TLSv1.3 in text@eqsin - T170567 |
[production] |
| 12:03 |
<jbond42> |
upgrade haproxy on dns servers |
[production] |
| 11:08 |
<Urbanecm> |
EU SWAT done |
[production] |
| 11:05 |
<Urbanecm> |
Purge https://en.wikipedia.org/static/images/project-logos/cswiki*.png (T249173) |
[production] |
| 11:04 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: SWAT: 7da408e: Revert "Enable cswiki anniversary logo" (T249173) (duration: 01m 00s) |
[production] |
| 11:01 |
<jynus> |
resizing backup1001:/srv/databases to 40 TB |
[production] |
| 10:55 |
<XioNoX> |
set uRPF log action to syslog infra wide - T244147 |
[production] |
| 10:15 |
<XioNoX> |
update prefix-list LVS-service-ips to add missing prefixes |
[production] |
| 09:49 |
<XioNoX> |
re-order aggregate routes to standardize order |
[production] |
| 09:48 |
<XioNoX> |
cleanup 2620:0:860::/46 and 208.80.152.0/22 aggregates from cr2-eqdfw - T246721 |
[production] |
| 09:47 |
<XioNoX> |
cleanup 2620:0:860::/46 and 208.80.152.0/22 aggregates from cr2-eqord - T246721 |
[production] |
| 09:37 |
<XioNoX> |
cleanup 2620:0:860::/46 and 208.80.152.0/22 aggregates from cr1/2-codfw - T246721 |
[production] |
| 09:16 |
<XioNoX> |
add missing `routing-options rib inet6.0 aggregate defaults discard` where missing (cr3-knams, cr3-esams, cr2-eqord, cr2-eqdfw, cr1/2-eqiad/codfw) |
[production] |
| 09:13 |
<godog> |
add mwilliams to 'wmf' ldap group - T249844 |
[production] |
| 09:08 |
<marostegui> |
Add kormat to ops and wmf ldap groups - T250134 |
[production] |
| 08:49 |
<elukey> |
restart elastic-chi on cloudelastic1001 with -XX:NewSize=10G - T231517 |
[production] |
| 08:05 |
<sonic2020> |
Upgrading to ATS 8.0.7, stage at 50% completed. |
[production] |
| 08:04 |
<sonic2020> |
joaquin: Upgrading to ATS 8.0.7, stage at 0% completed. |
[production] |
| 07:33 |
<elukey> |
apply CMS GC settings to chi on cloudelastic1001 - T231517 |
[production] |
| 05:30 |
<vgutierrez> |
rolling upgrade to ats 8.0.7-rc0-1wm1 in esams and eqiad |
[production] |
| 05:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool pc2008 after upgrade (duration: 01m 00s) |
[production] |
|
2020-04-13
§
|
| 23:24 |
<mdholloway> |
re-ran extensions/MachineVision/maintenance/withholdImages.php on commonswiki |
[production] |
| 23:14 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: MachineVision withholding list additions (T249939) (duration: 00m 59s) |
[production] |
| 22:41 |
<cdanis> |
repool codfw |
[production] |
| 22:35 |
<ebernhardson> |
restart elasticsearch_6@production-search-psi-eqiad on elastic1052 for excessive old gc over last few hours |
[production] |
| 22:35 |
<ebernhardson> |
restart elasticsearch_6@production-search-psi-eqiad on elastic1052 |
[production] |
| 22:08 |
<cdanis> |
depool codfw |
[production] |
| 21:43 |
<mdholloway> |
ran extensions/MachineVision/maintenance/removeBlacklistedSuggestions.php on commonswiki (T249273) |
[production] |
| 21:34 |
<mdholloway> |
ran extensions/MachineVision/maintenance/removeBlacklistedSuggestions.php on testcommonswiki |
[production] |
| 21:32 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.27/extensions/MachineVision: Add script to apply blacklist to current labels (T249273) (duration: 00m 58s) |
[production] |
| 20:49 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: MachineVision blocklist update (T249895) (duration: 00m 59s) |
[production] |
| 19:56 |
<mdholloway> |
finished running extensions/MachineVision/maintenance/withholdImages.php on commonswiki (T249939) |
[production] |
| 19:51 |
<mdholloway> |
running extensions/MachineVision/maintenance/withholdImages.php on commonswiki |
[production] |
| 19:41 |
<mdholloway> |
ran extensions/MachineVision/maintenance/withholdImages.php on testcommonswiki |
[production] |
| 19:37 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.27/extensions/MachineVision: Add support for WITHHOLD_ALL review state (T249939) (duration: 01m 23s) |
[production] |
| 19:13 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: MachineVision: Add MachineVisionWithholdImageList config (T249939) (duration: 01m 03s) |
[production] |
| 19:06 |
<niedzielski> |
Morning SWAT done |
[production] |