551-600 of 10000 results (18ms)
2020-05-27 ยง
14:07 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . [production]
14:07 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . [production]
14:04 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
14:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1146:3312, db1146:3314 and db1103:3312, db1103:3314', diff saved to https://phabricator.wikimedia.org/P11317 and previous config saved to /var/cache/conftool/dbconfig/20200527-140442-marostegui.json [production]
14:03 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
14:03 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
13:58 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
13:51 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
13:51 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
13:48 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
13:47 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
13:47 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
13:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1146:3312, db1146:3314 and db1103:3312, db1103:3314', diff saved to https://phabricator.wikimedia.org/P11316 and previous config saved to /var/cache/conftool/dbconfig/20200527-134704-marostegui.json [production]
13:45 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
13:42 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
13:36 <gehel@cumin1001> START - Cookbook sre.postgresql.postgres-init [production]
13:34 <gehel@cumin1001> END (FAIL) - Cookbook sre.postgresql.postgres-init (exit_code=99) [production]
13:34 <gehel@cumin1001> START - Cookbook sre.postgresql.postgres-init [production]
13:21 <gehel> repool maps2004 / depool maps2003 [production]
13:21 <ema> cp: upgrade purged to 0.14 [production]
13:18 <marostegui> Kill /usr/local/bin/mwscriptwikiset updateSpecialPages.php s8.dblist --override --only=Fewestrevisions T238199 [production]
13:16 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
13:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1146:3312, db1146:3314 and db1103:3312, db1103:3314', diff saved to https://phabricator.wikimedia.org/P11313 and previous config saved to /var/cache/conftool/dbconfig/20200527-131515-marostegui.json [production]
13:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1146:3312 and db1146:3314 to dbctl T252512', diff saved to https://phabricator.wikimedia.org/P11312 and previous config saved to /var/cache/conftool/dbconfig/20200527-130820-marostegui.json [production]
13:06 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
11:34 <Urbanecm> EU SWAT done [production]
11:28 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.34/extensions/GrowthExperiments/: SWAT: 983eda5: Mentorship dialog: Swap panel to ask-help on open (T253692) (duration: 01m 06s) [production]
11:18 <ema> cp2027: upgrade purged to 0.14 [production]
11:17 <ema> purged 0.14 uploaded to buster-wikimedia [production]
11:09 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|598678|Enable ContentTranslation in Galician Wikipedia as a default tool (T250355)]] (duration: 01m 18s) [production]
10:15 <hashar> contint2001: starting zuul [production]
10:15 <hashar> contint2001: started jenkins [production]
10:03 <mutante> contint2001 - find /var/lib/jenkins -user statsite -exec chown -h jenkins:jenkins {} \; [production]
10:02 <mutante> repeated rsync of /var/lib/jenkins with -p ; find /var/lib/jenkins -group bacula -user statsite -exec chown -h jenkins:jenkins {} \; [production]
09:55 <hashar> contint2001: starting jenkins [production]
09:54 <hashar> contint1001 / contint2001 : deleted obsolete files /var/lib/jenkins/.git and /var/lib/jenkins/jobs/_shared/ [production]
09:52 <mutante> contint2001 - find /var/lib/jenkins -user statsite -exec chown -h jenkins:jenkins {} \; [production]
09:51 <godog> roll restart prometheus on the fleet to apply I0e2fe8af [production]
09:49 <mutante> contint2001 - find /var/lib/jenkins -group bacula -user statsite -exec chown jenkins:jenkins {} \; [production]
09:48 <hashar> contint2001: unmasked jenkins and started it [production]
09:42 <filippo@cumin1001> conftool action : set/pooled=yes; selector: name=prometheus2003.codfw.wmnet [production]
09:42 <mutante> switching CI backend from contint1001 to contint2001 [production]
09:40 <mutante> repeated rsync -avp --delete /var/lib/zuul/ rsync://contint2001.wikimedia.org/ci--var-lib-zuul- [production]
09:40 <hashar> contint1001: masked jenkins and zuul [production]
09:39 <mutante> repeated rsync -avp --delete /var/lib/jenkins/ rsync://contint2001.wikimedia.org/ci--var-lib-jenkins- [production]
09:39 <hashar> Stopping Zuul and Jenkins CI for scheduled maintenance # T224591 [production]
09:35 <filippo@cumin1001> conftool action : set/pooled=no; selector: name=prometheus2003.codfw.wmnet [production]
08:52 <hashar> contint1001: find /srv/jenkins/builds/operations-puppet-wmf-style-guide -type f -name '*.tmp' -delete # T253729 [production]
08:48 <marostegui> Stop MySQL on db1103 [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1103:3312 db1103:3314 to clone db1146 T252512', diff saved to https://phabricator.wikimedia.org/P11308 and previous config saved to /var/cache/conftool/dbconfig/20200527-084713-marostegui.json [production]