1751-1800 of 10000 results (68ms)
2020-04-07 ยง
19:45 <hoo> Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata RDF dumps start now (broke as a side effect of T249565) [production]
19:13 <XioNoX> push pfw firewall rules - T249650 [production]
19:08 <jhuneidi@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.27 [production]
18:48 <jhuneidi@deploy1001> Pruned MediaWiki: 1.35.0-wmf.24 (duration: 12m 44s) [production]
17:56 <herron> increasing codfw.mediawiki.job.cirrusSearchElasticaWrite to 3 partitions T240702 [production]
17:55 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) retry (duration: 01m 02s) [production]
17:54 <addshore> last sync stuck on sync-masters [production]
17:54 <addshore@deploy1001> sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) (duration: 01m 16s) [production]
17:49 <ppchelko@deploy1001> Started restart [cpjobqueue/deploy@83c93d1]: Try to make it notice new partitions T240702 [production]
17:40 <herron> increasing eqiad.mediawiki.job.cirrusSearchElasticaWrite to 3 partitions T240702 [production]
16:24 <longma> 1.35.0-wmf.27 was branched at e76ac29cd9c57bed4097ec8a4ea8311fb55fd967 for T247774 [production]
16:16 <hashar> restarting CI jenkins [production]
15:53 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:21 <moritzm> installing idp-test2001 [production]
15:20 <XioNoX> enable uRPF loose mode (log only) on cr4-ulsfo - T244147 [production]
15:17 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (12/14.5h) (duration: 01m 00s) [production]
15:10 <ema> cp3052: stop purged, start vhtcpd T249583 T241232 [production]
15:00 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
14:56 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (10/14.5h) (duration: 00m 55s) [production]
14:52 <jeh> cloudvirt2003-dev: downtime in icinga and reboot to enable BIOS virtualization support T249453 [production]
14:38 <ema> cp3052: stop vhtcpd, start purged T249583 [production]
14:35 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (8/14.5h) (duration: 00m 58s) [production]
14:25 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (4/14.5h) (duration: 00m 58s) [production]
14:15 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (2/14.5h) (duration: 00m 58s) [production]
14:08 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) take 2 (duration: 00m 57s) [production]
13:57 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: REVERT T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 58s) [production]
13:55 <addshore@deploy1001> sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 29s) [production]
13:17 <vgutierrez> restart ats-tls on cp3056 - T249335 [production]
12:59 <vgutierrez> restart ats-tls on cp3052- T249335 [production]
12:50 <addshore> addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php --wiki=wikidatawiki --file T249596-6.list > T249596-6.out # T249565 [production]
12:42 <addshore> addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php --wiki=wikidatawiki --file T249596-5.list > T249596-5.out # T249565 [production]
12:42 <vgutierrez> restart ats-tls on cp3058 - T249335 [production]
12:25 <jmm@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
12:06 <addshore> addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php --wiki=wikidatawiki --file T249596-4.list > T249596-4.out # T249565 T249596 [production]
12:05 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
11:52 <marostegui@cumin1001> dbctl commit (dc=all): 'repool db1126', diff saved to https://phabricator.wikimedia.org/P10932 and previous config saved to /var/cache/conftool/dbconfig/20200407-115228-marostegui.json [production]
11:51 <marostegui@cumin1001> dbctl commit (dc=all): 'depool db1126', diff saved to https://phabricator.wikimedia.org/P10931 and previous config saved to /var/cache/conftool/dbconfig/20200407-115154-marostegui.json [production]
11:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1092, db1111, db1099:3318 after table rename', diff saved to https://phabricator.wikimedia.org/P10930 and previous config saved to /var/cache/conftool/dbconfig/20200407-115058-marostegui.json [production]
11:50 <jynus> renaming wb_items_per_site_recovered to wb_items_per_site on s8 [production]
11:45 <jynus> stopping s8 replication on db1116:3318, db1095:3318, db2079 [production]
11:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092, db1111, db1099:3318 for table rename', diff saved to https://phabricator.wikimedia.org/P10929 and previous config saved to /var/cache/conftool/dbconfig/20200407-114258-marostegui.json [production]
11:36 <Amir1> stopped the rebuilt script (T249565) [production]
11:34 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: cleanup T203888, Remove old unused RejectParserCacheValue hook (duration: 00m 59s) [production]
11:09 <marostegui> Deploy schema change on s3 codfw [production]
11:07 <jynus> starting recovery on all s8 hosts [production]
10:45 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
10:41 <addshore@deploy1001> Synchronized php-1.35.0-wmf.26/extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php: T249565 T249596 Wikibase rebuildItemsPerSite.php script that allows lists of ids (duration: 01m 00s) [production]
10:27 <jynus> starting recovery on db1099:3318 [production]
09:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1119 after schema change', diff saved to https://phabricator.wikimedia.org/P10927 and previous config saved to /var/cache/conftool/dbconfig/20200407-095852-marostegui.json [production]
09:49 <volans@deploy1001> Finished deploy [homer/deploy@887544c]: Release v0.2.0 (take 2) (duration: 00m 26s) [production]