3251-3300 of 8505 results (20ms)
2021-03-01 §
21:49 <mutante> deploy1002 - removed scap-global-lock, unlocked scap [production]
21:18 <mutante> mw1262 - running puppet to switch to new deployment server, scap pull [production]
21:05 <mutante> re-enabling puppet on deploy1001 - running puppet on deploy*, switching eqiad scap master and deployment_server globally (T265963) [production]
20:37 <mutante> deploy1001 - disable puppet and manually create scap-global-lock - NO DEPLOYMENTS [production]
2021-02-25 §
23:55 <mutante> deploy1002, deploy2002 - scap-master-sync deploy1001.eqiad.wmnet (T265963) [production]
23:39 <mutante> deploy2001 - scap-master-sync from deploy1001 runs and attempts to --delete files to stay in sync but fails to do so because *.cdb files are in cache dirs and rsync does not want to delete non-empty directories, this leads to build up of the size of /srv/mediawiki-staging to 10 times the size of eqiad [production]
23:34 <mutante> deploy2001 - scap-master-sync from deploy1001 [production]
23:13 <mutante> deploy1002 - /usr/local/bin/scap-master-sync deploy1001.eqiad.wmnet [production]
2021-02-24 §
09:06 <volans> run "sudo find . -user root -exec chown netbox. '{}' \;" in /srv/deployment/netbox/deploy-cache/revs on netbox* hosts to prevent scap failures on cleanup - T265084 [production]
2021-02-23 §
21:28 <jhuneidi@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.32 refs T274936 (duration: 36m 52s) [production]
20:52 <jhuneidi@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.32 refs T274936 [production]
2021-02-22 §
16:51 <Urbanecm> Run scap pull on mwmaint1002 to clear any local changes [production]
2021-02-19 §
20:33 <mutante> mw1261, mw1270 - scap pull [production]
20:32 <mutante> mw1287 - scap pull [production]
01:22 <mutante> mwmaint2001 back on buster and back in scap dsh groups (if anything pops up you can revert 665175) [production]
00:54 <mutante> mwmaint2001 - back from reimage - scap pull [production]
2021-02-16 §
20:20 <mutante> mwdebug1002 has been recreated on buster and has been repooled after scap pull - you can find a .tar.gz in your home with the contents of your home before reimaging, fingerprint at T274023#6835116 [production]
18:49 <dduvall@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.31 (duration: 49m 37s) [production]
18:04 <dduvall@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.31 [production]
2021-02-12 §
20:32 <mutante> mw1353, mw1358 - scap pull, repooled [production]
2021-02-11 §
22:57 <Urbanecm> Run scap pull at mwmaint1002 [production]
2021-02-10 §
00:22 <twentyafterfour@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.30 (duration: 24m 10s) [production]
2021-02-09 §
23:58 <twentyafterfour@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.30 [production]
23:55 <twentyafterfour@deploy1001> Finished scap: (no justification provided) (duration: 08m 43s) [production]
23:50 <mutante> mw1383,mw1385 - scap pull, php [production]
23:47 <twentyafterfour> running scap sync-world [production]
23:47 <twentyafterfour@deploy1001> Started scap: (no justification provided) [production]
07:47 <hashar@deploy1001> Finished deploy [integration/docroot@672e79f]: build: Add /scap/log to gitignore (duration: 00m 06s) [production]
07:47 <hashar@deploy1001> Started deploy [integration/docroot@672e79f]: build: Add /scap/log to gitignore [production]
2021-02-05 §
19:39 <mutante> reimaging 2 scap proxies in codfw because there are no deployments today [production]
2021-02-02 §
23:53 <mutante> mw1300 - scap pull (it crashed earlier put is back after powercycling) [production]
14:26 <hashar@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.29 (duration: 73m 10s) [production]
13:13 <hashar@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.29 [production]
2021-02-01 §
14:12 <ladsgroup@deploy1001> Finished scap: [[gerrit:660796|Add Multilingual Wikisource to list of Wikidata's special sites]] (T138332) (duration: 21m 52s) [production]
13:50 <ladsgroup@deploy1001> Started scap: [[gerrit:660796|Add Multilingual Wikisource to list of Wikidata's special sites]] (T138332) [production]
13:47 <ladsgroup@deploy1001> scap sync-l10n completed (1.36.0-wmf.28) (duration: 00m 58s) [production]
10:54 <hashar@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.28 (duration: 07m 48s) [production]
10:46 <hashar@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.28 [production]
2021-01-29 §
22:36 <dancy@deploy1001> Finished scap: MW servers complaining about l10n files after .27 rollback (duration: 07m 22s) [production]
22:29 <dancy@deploy1001> Started scap: MW servers complaining about l10n files after .27 rollback [production]
22:09 <dancy@deploy1001> scap failed: average error rate on 8/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) [production]
2021-01-28 §
20:13 <mutante> scap pulling and repooling: mw1264, mw2223, mw2247 [production]
11:01 <_joe_> restarting php-fpm on the appserver,api and jobrunner clusters in eqiad, 10% at a time, for simulating scap rolling restarts T266055 [production]
10:33 <_joe_> performing a test-run of the rolling restart of php-fpm in codfw, using the same code scap will use T266055. Starting from the api cluster, then proceeding whith others [production]
00:12 <urbanecm@deploy1001> Scap failed!: Call to mwscript eval.php stderr: not empty [production]
2021-01-26 §
18:46 <dancy@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.28 (duration: 40m 09s) [production]
18:07 <dancy@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.28 [production]
2021-01-20 §
23:01 <mutante> mw2331, mw2333 - scap pull [production]
2021-01-19 §
18:42 <brennen@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.27 (duration: 41m 57s) [production]
18:01 <brennen@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.27 [production]