4501-4550 of 10000 results (44ms)
2021-04-15 §
08:48 <godog> free space and bounce thanos-compact [production]
08:36 <marostegui@cumin1001> dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 10%: Repool db1146:3314 after kernel upgrade', diff saved to https://phabricator.wikimedia.org/P15359 and previous config saved to /var/cache/conftool/dbconfig/20210415-083618-root.json [production]
08:21 <marostegui@cumin1001> dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 5%: Repool db1146:3314 after kernel upgrade', diff saved to https://phabricator.wikimedia.org/P15358 and previous config saved to /var/cache/conftool/dbconfig/20210415-082115-root.json [production]
08:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1146:3314', diff saved to https://phabricator.wikimedia.org/P15357 and previous config saved to /var/cache/conftool/dbconfig/20210415-081127-marostegui.json [production]
08:09 <marostegui@cumin1001> dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Repool db1146:3314 after kernel upgrade', diff saved to https://phabricator.wikimedia.org/P15356 and previous config saved to /var/cache/conftool/dbconfig/20210415-080947-root.json [production]
07:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: Repool db1166 after cloning db1179', diff saved to https://phabricator.wikimedia.org/P15355 and previous config saved to /var/cache/conftool/dbconfig/20210415-075718-root.json [production]
07:54 <marostegui@cumin1001> dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 75%: Repool db1146:3314 after kernel upgrade', diff saved to https://phabricator.wikimedia.org/P15354 and previous config saved to /var/cache/conftool/dbconfig/20210415-075444-root.json [production]
07:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: Repool db1166 after cloning db1179', diff saved to https://phabricator.wikimedia.org/P15353 and previous config saved to /var/cache/conftool/dbconfig/20210415-074214-root.json [production]
07:39 <marostegui@cumin1001> dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 50%: Repool db1146:3314 after kernel upgrade', diff saved to https://phabricator.wikimedia.org/P15352 and previous config saved to /var/cache/conftool/dbconfig/20210415-073940-root.json [production]
07:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Repool db1166 after cloning db1179', diff saved to https://phabricator.wikimedia.org/P15351 and previous config saved to /var/cache/conftool/dbconfig/20210415-072711-root.json [production]
07:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 25%: Repool db1146:3314 after kernel upgrade', diff saved to https://phabricator.wikimedia.org/P15350 and previous config saved to /var/cache/conftool/dbconfig/20210415-072436-root.json [production]
07:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1146 (s2,s4) to upgrade kernel', diff saved to https://phabricator.wikimedia.org/P15348 and previous config saved to /var/cache/conftool/dbconfig/20210415-071600-marostegui.json [production]
07:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Repool db1166 after cloning db1179', diff saved to https://phabricator.wikimedia.org/P15347 and previous config saved to /var/cache/conftool/dbconfig/20210415-071207-root.json [production]
06:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 10%: Repool db1166 after cloning db1179', diff saved to https://phabricator.wikimedia.org/P15346 and previous config saved to /var/cache/conftool/dbconfig/20210415-065704-root.json [production]
06:33 <ryankemper> !log T280108 T267927 `data-transfer` to `wdqs1004` was successful; cookbook failed due to a newly introduced minor type error that didn't effect the transfer itself [production]
06:32 <elukey> move hue.wikimedia.org to an-tool1009 (from analytics-tool1001) [production]
06:00 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
05:54 <Amir1> end of cleaning archive of pywikibot-bugs and wikidata-bugs T262773 [production]
05:44 <Amir1> start deleting archive of wikidata-bugs T262773 [production]
05:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1166 to clone db1179 T275633', diff saved to https://phabricator.wikimedia.org/P15344 and previous config saved to /var/cache/conftool/dbconfig/20210415-050239-marostegui.json [production]
04:14 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
04:14 <ryankemper> T280108 T267927 `wdqs2008` (source) caught up on lag, xfering to `wdqs1004`: `sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1004.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927` [production]
04:06 <ryankemper> T280108 T267927 Merged https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/679320, will verify correct behavior of `data-transfer` cookbook [production]
01:19 <Amir1> mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P8671 --new-data-type external-id (T278427) [production]
00:50 <ejegg> updated fundraising CiviCRM from c3342aa4ea to 35a8dd33ba [production]
2021-04-14 §
23:27 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:679350|Disable legacy javascript global variables in ruwiki (T72470)]] (duration: 01m 16s) [production]
21:44 <legoktm> manually started debmonitor-client.service on ml-serve2004 after 502 Bad gateway error [production]
20:55 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wtp[1037-1039].eqiad.wmnet with reason: reimage [production]
20:54 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wtp[1037-1039].eqiad.wmnet with reason: reimage [production]
20:38 <mutante> wtp1037, wtp1038, wtp1039 - scap pull [production]
19:52 <dzahn@cumin1001> conftool action : set/weight=20; selector: name=mw2395.codfw.wmnet,cluster=jobrunner [production]
19:52 <dzahn@cumin1001> conftool action : set/weight=20; selector: name=mw2394.codfw.wmnet,cluster=jobrunner [production]
19:51 <dzahn@cumin1001> conftool action : set/weight=20; selector: name=mw2410.codfw.wmnet,cluster=videoscaler [production]
19:51 <dzahn@cumin1001> conftool action : set/weight=20; selector: name=mw2411.codfw.wmnet,cluster=videoscaler [production]
19:50 <cstone> civicrm revision changed from ec2a3bcff6 to c3342aa4ea [production]
19:50 <dzahn@cumin1001> conftool action : set/weight=15; selector: name=mw2411.codfw.wmnet,cluster=videoscaler [production]
19:50 <dzahn@cumin1001> conftool action : set/weight=15; selector: name=mw2410.codfw.wmnet,cluster=videoscaler [production]
19:49 <dzahn@cumin1001> conftool action : set/weight=10; selector: name=mw2395.codfw.wmnet,cluster=videoscaler [production]
19:48 <dzahn@cumin1001> conftool action : set/weight=10; selector: name=mw2394.codfw.wmnet,cluster=videoscaler [production]
19:47 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2411.codfw.wmnet,cluster=jobrunner [production]
19:42 <herron> migrating kafka-logging broker logstash1011 to kafka-logging1002 T279342 [production]
19:06 <jhuneidi@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.1 refs T278345 (duration: 02m 03s) [production]
19:04 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.1 refs T278345 [production]
18:58 <mutante> urldownloader1002 - icinga alerted about disk space, ran 'apt-get clean' which is my usual go to in that case. it reduced usage from 97% to 89% [production]
17:56 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.1/extensions/GrowthExperiments/: ce44792: 84107c5: GrowthExperiments backports related to DatabaseMentorStore (T279957; T279959) (duration: 01m 55s) [production]
15:00 <shdubsh> run new curator actions on codfw - T274394 [production]
14:48 <shdubsh> O:logstash::elasticsearch7 update elasticsearch-curator to 5.8.1 [production]
14:13 <rzl> mcrouter cert renewal complete, puppet re-enabled T276029 [production]
14:11 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@8ae53e3]: T273847 export queries to relforge dag deployment - start date update (duration: 02m 14s) [production]
14:11 <moritzm> installing intel-microcode updates on Buster [production]