|
2021-12-03
§
|
| 06:20 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1144.eqiad.wmnet with reason: Maintenance T277354 |
[production] |
| 06:20 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db1144.eqiad.wmnet with reason: Maintenance T277354 |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db1113:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P18004 and previous config saved to /var/cache/conftool/dbconfig/20211203-062011-marostegui.json |
[production] |
| 06:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P18003 and previous config saved to /var/cache/conftool/dbconfig/20211203-060506-marostegui.json |
[production] |
| 06:02 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1125.eqiad.wmnet with OS bullseye |
[production] |
| 05:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P18002 and previous config saved to /var/cache/conftool/dbconfig/20211203-055001-marostegui.json |
[production] |
| 05:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db1113:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P18001 and previous config saved to /var/cache/conftool/dbconfig/20211203-053457-marostegui.json |
[production] |
| 05:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1113:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P18000 and previous config saved to /var/cache/conftool/dbconfig/20211203-053032-marostegui.json |
[production] |
| 05:30 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1113.eqiad.wmnet with reason: Maintenance T277354 |
[production] |
| 05:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db1113.eqiad.wmnet with reason: Maintenance T277354 |
[production] |
| 01:39 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2025.codfw.wmnet with OS buster |
[production] |
| 01:06 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host restbase2025.codfw.wmnet with OS buster |
[production] |
| 01:05 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2024.codfw.wmnet with OS buster |
[production] |
| 01:01 |
<tgr> |
UTC late deploys done |
[production] |
| 01:00 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments: Backport: [[gerrit:743177|Add an image: Add test version of GEInfoboxTemplates (T291232)]] (duration: 00m 57s) |
[production] |
| 00:44 |
<rzl> |
rzl@apt1001:~$ sudo -i reprepro -C main include buster-wikimedia /home/rzl/python3-imagecatalog/imagecatalog_0.0.1-1_amd64.changes |
[production] |
| 00:37 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes: Backport: [[gerrit:743178|Avoid references to TemplateCollectionFeature]] step2 (duration: 00m 56s) |
[production] |
| 00:36 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes/Config/Validation/GrowthConfigValidation.php: Backport: [[gerrit:743178|Avoid references to TemplateCollectionFeature]] step 1 (duration: 00m 56s) |
[production] |
| 00:33 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host restbase2024.codfw.wmnet with OS buster |
[production] |
|
2021-12-02
§
|
| 20:05 |
<legoktm> |
re-pooling mw1414 following testing |
[production] |
| 19:35 |
<legoktm> |
installing yaml PHP extension on canaries |
[production] |
| 19:29 |
<andrewbogott> |
upgrading wikitech-static deb packages as well as moving to mediawiki 1.37.0 |
[production] |
| 19:26 |
<majavah> |
UTC evening deploys done |
[production] |
| 19:26 |
<taavi@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/WikimediaEvents/modules/ext.wikimediaEvents/webUIScroll.js: Backport: [[gerrit:743227|Update scroll instrument (T294246)]] (duration: 00m 56s) |
[production] |
| 19:22 |
<taavi@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:720363|Drop old config names for CentralAuth denylist controls (T277932)]] (duration: 00m 56s) |
[production] |
| 19:12 |
<taavi@deploy1002> |
Synchronized wmf-config: Config: [[gerrit:739032|GrowthExperiments configuration fixes (T294737)]] (duration: 00m 57s) |
[production] |
| 18:56 |
<legoktm> |
upgraded scap to 4.1.0 on A:mw-canary, A:parsoid-canary, A:mw-jobrunner-canary (T296867) |
[production] |
| 18:45 |
<legoktm> |
uploaded scap 4.1.0 to apt.wm.o (T296867) |
[production] |
| 18:22 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 18:19 |
<vgutierrez> |
re-enable puppet on cp3064 - T296874 |
[production] |
| 18:14 |
<hoo> |
Started Wikibase rebuildItemsPerSite on mwmaint1002 for wikidatawiki. Can be killed at any time, if necessary. |
[production] |
| 17:51 |
<vgutierrez> |
puppet disabled on cp3064 to manually increase number of maxconns in HAProxy - T296874 |
[production] |
| 17:38 |
<ryankemper> |
[WDQS] Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/743216/; as a result of the fix `'-Dwdqs.throttling-filter.time-bucket-capacity-in-seconds=240', '-Dwdqs.throttling-filter.time-bucket-refill-amount-in-seconds=120', '-Dwdqs.throttling-filter.ban-duration-in-minutes=60'` will now be in the `extra_jvm_opts` for `wdqs-internal` hosts |
[production] |
| 15:38 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2022.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage |
[production] |
| 15:38 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2022.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage |
[production] |
| 14:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17997 and previous config saved to /var/cache/conftool/dbconfig/20211202-145151-marostegui.json |
[production] |
| 14:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17996 and previous config saved to /var/cache/conftool/dbconfig/20211202-143646-marostegui.json |
[production] |
| 14:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17995 and previous config saved to /var/cache/conftool/dbconfig/20211202-142141-marostegui.json |
[production] |
| 14:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17994 and previous config saved to /var/cache/conftool/dbconfig/20211202-140636-marostegui.json |
[production] |
| 14:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17993 and previous config saved to /var/cache/conftool/dbconfig/20211202-140557-marostegui.json |
[production] |
| 14:05 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2137.codfw.wmnet with reason: Maintenance T277354 |
[production] |
| 14:05 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2137.codfw.wmnet with reason: Maintenance T277354 |
[production] |
| 14:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17992 and previous config saved to /var/cache/conftool/dbconfig/20211202-140548-marostegui.json |
[production] |
| 13:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17990 and previous config saved to /var/cache/conftool/dbconfig/20211202-135043-marostegui.json |
[production] |
| 13:49 |
<hnowlan> |
roll-restarting tilerator,tileratorui,kartotherian in eqiad |
[production] |
| 13:37 |
<hnowlan> |
roll-restarting tilerator,tileratorui,kartotherian in codfw |
[production] |
| 13:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17989 and previous config saved to /var/cache/conftool/dbconfig/20211202-133538-marostegui.json |
[production] |
| 13:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17988 and previous config saved to /var/cache/conftool/dbconfig/20211202-132034-marostegui.json |
[production] |
| 13:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17987 and previous config saved to /var/cache/conftool/dbconfig/20211202-131959-marostegui.json |
[production] |
| 13:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2094,2128].codfw.wmnet with reason: Maintenance T277354 |
[production] |