1851-1900 of 10000 results (42ms)
2021-12-03 §
01:39 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2025.codfw.wmnet with OS buster [production]
01:06 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host restbase2025.codfw.wmnet with OS buster [production]
01:05 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2024.codfw.wmnet with OS buster [production]
01:01 <tgr> UTC late deploys done [production]
01:00 <tgr@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments: Backport: [[gerrit:743177|Add an image: Add test version of GEInfoboxTemplates (T291232)]] (duration: 00m 57s) [production]
00:44 <rzl> rzl@apt1001:~$ sudo -i reprepro -C main include buster-wikimedia /home/rzl/python3-imagecatalog/imagecatalog_0.0.1-1_amd64.changes [production]
00:37 <tgr@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes: Backport: [[gerrit:743178|Avoid references to TemplateCollectionFeature]] step2 (duration: 00m 56s) [production]
00:36 <tgr@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes/Config/Validation/GrowthConfigValidation.php: Backport: [[gerrit:743178|Avoid references to TemplateCollectionFeature]] step 1 (duration: 00m 56s) [production]
00:33 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host restbase2024.codfw.wmnet with OS buster [production]
2021-12-02 §
20:05 <legoktm> re-pooling mw1414 following testing [production]
19:35 <legoktm> installing yaml PHP extension on canaries [production]
19:29 <andrewbogott> upgrading wikitech-static deb packages as well as moving to mediawiki 1.37.0 [production]
19:26 <majavah> UTC evening deploys done [production]
19:26 <taavi@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/WikimediaEvents/modules/ext.wikimediaEvents/webUIScroll.js: Backport: [[gerrit:743227|Update scroll instrument (T294246)]] (duration: 00m 56s) [production]
19:22 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:720363|Drop old config names for CentralAuth denylist controls (T277932)]] (duration: 00m 56s) [production]
19:12 <taavi@deploy1002> Synchronized wmf-config: Config: [[gerrit:739032|GrowthExperiments configuration fixes (T294737)]] (duration: 00m 57s) [production]
18:56 <legoktm> upgraded scap to 4.1.0 on A:mw-canary, A:parsoid-canary, A:mw-jobrunner-canary (T296867) [production]
18:45 <legoktm> uploaded scap 4.1.0 to apt.wm.o (T296867) [production]
18:22 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
18:19 <vgutierrez> re-enable puppet on cp3064 - T296874 [production]
18:14 <hoo> Started Wikibase rebuildItemsPerSite on mwmaint1002 for wikidatawiki. Can be killed at any time, if necessary. [production]
17:51 <vgutierrez> puppet disabled on cp3064 to manually increase number of maxconns in HAProxy - T296874 [production]
17:38 <ryankemper> [WDQS] Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/743216/; as a result of the fix `'-Dwdqs.throttling-filter.time-bucket-capacity-in-seconds=240', '-Dwdqs.throttling-filter.time-bucket-refill-amount-in-seconds=120', '-Dwdqs.throttling-filter.ban-duration-in-minutes=60'` will now be in the `extra_jvm_opts` for `wdqs-internal` hosts [production]
15:38 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2022.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage [production]
15:38 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2022.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage [production]
14:51 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17997 and previous config saved to /var/cache/conftool/dbconfig/20211202-145151-marostegui.json [production]
14:36 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17996 and previous config saved to /var/cache/conftool/dbconfig/20211202-143646-marostegui.json [production]
14:21 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17995 and previous config saved to /var/cache/conftool/dbconfig/20211202-142141-marostegui.json [production]
14:06 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17994 and previous config saved to /var/cache/conftool/dbconfig/20211202-140636-marostegui.json [production]
14:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2137:3315 (T277354)', diff saved to https://phabricator.wikimedia.org/P17993 and previous config saved to /var/cache/conftool/dbconfig/20211202-140557-marostegui.json [production]
14:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2137.codfw.wmnet with reason: Maintenance T277354 [production]
14:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db2137.codfw.wmnet with reason: Maintenance T277354 [production]
14:05 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17992 and previous config saved to /var/cache/conftool/dbconfig/20211202-140548-marostegui.json [production]
13:50 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17990 and previous config saved to /var/cache/conftool/dbconfig/20211202-135043-marostegui.json [production]
13:49 <hnowlan> roll-restarting tilerator,tileratorui,kartotherian in eqiad [production]
13:37 <hnowlan> roll-restarting tilerator,tileratorui,kartotherian in codfw [production]
13:35 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17989 and previous config saved to /var/cache/conftool/dbconfig/20211202-133538-marostegui.json [production]
13:20 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17988 and previous config saved to /var/cache/conftool/dbconfig/20211202-132034-marostegui.json [production]
13:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2128 (T277354)', diff saved to https://phabricator.wikimedia.org/P17987 and previous config saved to /var/cache/conftool/dbconfig/20211202-131959-marostegui.json [production]
13:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2094,2128].codfw.wmnet with reason: Maintenance T277354 [production]
13:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2094,2128].codfw.wmnet with reason: Maintenance T277354 [production]
13:19 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17986 and previous config saved to /var/cache/conftool/dbconfig/20211202-131949-marostegui.json [production]
13:04 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17985 and previous config saved to /var/cache/conftool/dbconfig/20211202-130444-marostegui.json [production]
12:49 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17983 and previous config saved to /var/cache/conftool/dbconfig/20211202-124940-marostegui.json [production]
12:34 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17982 and previous config saved to /var/cache/conftool/dbconfig/20211202-123435-marostegui.json [production]
12:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2113 (T277354)', diff saved to https://phabricator.wikimedia.org/P17981 and previous config saved to /var/cache/conftool/dbconfig/20211202-123356-marostegui.json [production]
12:33 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2113.codfw.wmnet with reason: Maintenance T277354 [production]
12:33 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db2113.codfw.wmnet with reason: Maintenance T277354 [production]
12:33 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db2111 (T277354)', diff saved to https://phabricator.wikimedia.org/P17980 and previous config saved to /var/cache/conftool/dbconfig/20211202-123348-marostegui.json [production]
12:31 <moritzm> installing NSS security updates [production]