|
2020-04-07
§
|
| 20:39 |
<andrewbogott> |
correction: briefly downtiming ldap-eqiad-replica0 and ldap-eqiad-replica1. I'm trying to investigate a possible split-brain so going to turn ldap off on one, and then the other, to see if behavior changes |
[production] |
| 20:37 |
<andrewbogott> |
briefly downtiming serpens and seaborgium. I'm trying to investigate a possible split-brain so going to turn ldap off on one, and then the other, to see if behavior changes |
[production] |
| 20:34 |
<hoo> |
(Take 3) Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata RDF dumps start now (broke as a side effect of T249565) |
[production] |
| 20:17 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.27 refs T247774 |
[production] |
| 20:09 |
<jhuneidi@deploy1001> |
Finished scap: testwikis wikis to 1.35.0-wmf.27 (duration: 60m 34s) |
[production] |
| 20:08 |
<hoo> |
(Take 2) Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata RDF dumps start now (broke as a side effect of T249565) |
[production] |
| 19:45 |
<hoo> |
Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata RDF dumps start now (broke as a side effect of T249565) |
[production] |
| 19:13 |
<XioNoX> |
push pfw firewall rules - T249650 |
[production] |
| 19:08 |
<jhuneidi@deploy1001> |
Started scap: testwikis wikis to 1.35.0-wmf.27 |
[production] |
| 18:48 |
<jhuneidi@deploy1001> |
Pruned MediaWiki: 1.35.0-wmf.24 (duration: 12m 44s) |
[production] |
| 17:56 |
<herron> |
increasing codfw.mediawiki.job.cirrusSearchElasticaWrite to 3 partitions T240702 |
[production] |
| 17:55 |
<addshore@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) retry (duration: 01m 02s) |
[production] |
| 17:54 |
<addshore> |
last sync stuck on sync-masters |
[production] |
| 17:54 |
<addshore@deploy1001> |
sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) (duration: 01m 16s) |
[production] |
| 17:49 |
<ppchelko@deploy1001> |
Started restart [cpjobqueue/deploy@83c93d1]: Try to make it notice new partitions T240702 |
[production] |
| 17:40 |
<herron> |
increasing eqiad.mediawiki.job.cirrusSearchElasticaWrite to 3 partitions T240702 |
[production] |
| 16:24 |
<longma> |
1.35.0-wmf.27 was branched at e76ac29cd9c57bed4097ec8a4ea8311fb55fd967 for T247774 |
[production] |
| 16:16 |
<hashar> |
restarting CI jenkins |
[production] |
| 15:53 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 15:21 |
<moritzm> |
installing idp-test2001 |
[production] |
| 15:20 |
<XioNoX> |
enable uRPF loose mode (log only) on cr4-ulsfo - T244147 |
[production] |
| 15:17 |
<addshore@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (12/14.5h) (duration: 01m 00s) |
[production] |
| 15:10 |
<ema> |
cp3052: stop purged, start vhtcpd T249583 T241232 |
[production] |
| 15:00 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 14:56 |
<addshore@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (10/14.5h) (duration: 00m 55s) |
[production] |
| 14:52 |
<jeh> |
cloudvirt2003-dev: downtime in icinga and reboot to enable BIOS virtualization support T249453 |
[production] |
| 14:38 |
<ema> |
cp3052: stop vhtcpd, start purged T249583 |
[production] |
| 14:35 |
<addshore@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (8/14.5h) (duration: 00m 58s) |
[production] |
| 14:25 |
<addshore@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (4/14.5h) (duration: 00m 58s) |
[production] |
| 14:15 |
<addshore@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (2/14.5h) (duration: 00m 58s) |
[production] |
| 14:08 |
<addshore@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) take 2 (duration: 00m 57s) |
[production] |
| 13:57 |
<addshore@deploy1001> |
Synchronized wmf-config/CommonSettings.php: REVERT T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 58s) |
[production] |
| 13:55 |
<addshore@deploy1001> |
sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 29s) |
[production] |
| 13:17 |
<vgutierrez> |
restart ats-tls on cp3056 - T249335 |
[production] |
| 12:59 |
<vgutierrez> |
restart ats-tls on cp3052- T249335 |
[production] |
| 12:50 |
<addshore> |
addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php --wiki=wikidatawiki --file T249596-6.list > T249596-6.out # T249565 |
[production] |
| 12:42 |
<addshore> |
addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php --wiki=wikidatawiki --file T249596-5.list > T249596-5.out # T249565 |
[production] |
| 12:42 |
<vgutierrez> |
restart ats-tls on cp3058 - T249335 |
[production] |
| 12:25 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 12:06 |
<addshore> |
addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php --wiki=wikidatawiki --file T249596-4.list > T249596-4.out # T249565 T249596 |
[production] |
| 12:05 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 11:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'repool db1126', diff saved to https://phabricator.wikimedia.org/P10932 and previous config saved to /var/cache/conftool/dbconfig/20200407-115228-marostegui.json |
[production] |
| 11:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'depool db1126', diff saved to https://phabricator.wikimedia.org/P10931 and previous config saved to /var/cache/conftool/dbconfig/20200407-115154-marostegui.json |
[production] |
| 11:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1092, db1111, db1099:3318 after table rename', diff saved to https://phabricator.wikimedia.org/P10930 and previous config saved to /var/cache/conftool/dbconfig/20200407-115058-marostegui.json |
[production] |
| 11:50 |
<jynus> |
renaming wb_items_per_site_recovered to wb_items_per_site on s8 |
[production] |
| 11:45 |
<jynus> |
stopping s8 replication on db1116:3318, db1095:3318, db2079 |
[production] |
| 11:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092, db1111, db1099:3318 for table rename', diff saved to https://phabricator.wikimedia.org/P10929 and previous config saved to /var/cache/conftool/dbconfig/20200407-114258-marostegui.json |
[production] |
| 11:36 |
<Amir1> |
stopped the rebuilt script (T249565) |
[production] |
| 11:34 |
<addshore@deploy1001> |
Synchronized wmf-config/CommonSettings.php: cleanup T203888, Remove old unused RejectParserCacheValue hook (duration: 00m 59s) |
[production] |