1251-1300 of 10000 results (40ms)
2021-10-04 §
12:02 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti2025.codfw.wmnet with reason: Ganeti tests [production]
12:02 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti2025.codfw.wmnet with reason: Ganeti tests [production]
12:01 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:55 <urbanecm> EU B&C window done [production]
11:55 <urbanecm@deploy1002> Synchronized multiversion/MWWikiversions.php: 508cf5cc6d213373f7c9ba1cdef142ebc8398022: Let DB expressions intersect DB lists (T290609) (duration: 00m 58s) [production]
11:50 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: a855078cf52d88cc2cd27a0adc7c6a680c80dd39: dewiki, nlwiki: Bump Growth features to 80% (T288420, T285254) (duration: 00m 58s) [production]
11:46 <urbanecm@deploy1002> Synchronized private/PrivateSettings.php: 5728376: Update T250887 mitigations (duration: 00m 58s) [production]
11:44 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: b0a96bed4562bcc975187b1d34626201d407404b: Undeploy GettingStarted V: Remove now-obsolete logging channels (T235752) (duration: 00m 59s) [production]
11:42 <urbanecm@deploy1002> Synchronized wmf-config/extension-list: 9709bcfc8dacbcd1704471df08c31cec0711bea6: Undeploy GettingStarted IV: Dont build i18n (T235752) (duration: 00m 58s) [production]
11:39 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: d60f332785868797e7ecc9b5e410616d5604b392: Undeploy getting started III: Dont set wmgUseGettingStarted, now ignored (T235752) (duration: 00m 58s) [production]
11:37 <urbanecm@deploy1002> Synchronized wmf-config/CommonSettings.php: 9eaf960c4b7c304be57dfc8d248aca0c6501d04c: Undeploy GettingStarted II: Dont load regardless of config (T235752) (duration: 00m 58s) [production]
11:35 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 1c7405ad1eb323a8da524819f17d6f1a66afaa57: Undeploy GettingStarted I: Disable on all wikis (T235752) (duration: 00m 58s) [production]
11:31 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:724992|Remove deprecated SectionTranslationTargetLanguage config (T290302)]] (duration: 00m 58s) [production]
11:22 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:725042|Add wikisource-bot.toolforge.org to Commons copy upload list (T292213)]] (duration: 00m 59s) [production]
11:16 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:720058|Add IA-Upload tool domains to Commons wgCopyUploadsDomains (T287241)]] (duration: 00m 59s) [production]
11:12 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mathoid' for release 'production' . [production]
11:10 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mathoid' for release 'production' . [production]
11:07 <jiji@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:06 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mathoid' for release 'staging' . [production]
11:04 <effie> depool wtp1026 for tests [production]
11:04 <effie> pool wtp1025 [production]
10:59 <jiji@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
09:13 <akosiaris> hbal -L -G row_C -X on ganeti01.svc.eqiad.wmnet [production]
08:59 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@071f7c3] (eqiad): Increase mirrored traffic to 100% for eqiad (duration: 00m 54s) [production]
08:58 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@071f7c3] (eqiad): Increase mirrored traffic to 100% for eqiad [production]
07:37 <joal@deploy1002> Finished deploy [analytics/refinery@38f3adc] (hadoop-test): Hotfix analytics deploy TEST [analytics/refinery@38f3adc] (duration: 06m 14s) [production]
07:31 <joal@deploy1002> Started deploy [analytics/refinery@38f3adc] (hadoop-test): Hotfix analytics deploy TEST [analytics/refinery@38f3adc] [production]
07:30 <joal@deploy1002> Finished deploy [analytics/refinery@38f3adc] (thin): Hotfix analytics deploy THIN [analytics/refinery@38f3adc] (duration: 00m 06s) [production]
07:30 <joal@deploy1002> Started deploy [analytics/refinery@38f3adc] (thin): Hotfix analytics deploy THIN [analytics/refinery@38f3adc] [production]
07:29 <joal@deploy1002> Finished deploy [analytics/refinery@38f3adc]: Hotfix analytics deploy [analytics/refinery@38f3adc] (duration: 19m 18s) [production]
07:19 <dcausse> restarting blazegraph on wdqs2001 & wdqs2004 (allocators burning too quickly) [production]
07:18 <elukey> depool + restart blazegraph + restart updater for wdqs1006 [production]
07:18 <elukey@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=wdqs1006.wmnet [production]
07:18 <elukey@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=wdqs1004.wmnet [production]
07:10 <joal@deploy1002> Started deploy [analytics/refinery@38f3adc]: Hotfix analytics deploy [analytics/refinery@38f3adc] [production]
07:02 <godog> swift eqiad-prod: add weight to ms-be10[64-67] - T290546 [production]
06:44 <elukey> depool + restart blazegraph + restart updater on wdqs1004 [production]
05:50 <ladsgroup@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
05:49 <ladsgroup@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
05:47 <ladsgroup@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
2021-10-03 §
14:45 <_joe_> restarting acmechief on acmechief1001 [production]
12:55 <kormat@cumin1001> dbctl commit (dc=all): 'Depool db1127, bad ram', diff saved to https://phabricator.wikimedia.org/P17414 and previous config saved to /var/cache/conftool/dbconfig/20211003-125530-kormat.json [production]
08:24 <elukey> powercycle cp5006 (unresponsive to ssh, remote tty available but not able to login as root, no prometheus metrics in hours) [production]
08:23 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp5006.eqsin.wmnet [production]
2021-10-02 §
17:28 <bd808@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
16:10 <bd808@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
2021-10-01 §
23:19 <bd808@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
22:27 <mutante> puppetmaster2001 - systemctl reset-failed [production]
22:16 <mutante> puppetmaster2001 systemctl disable geoip_update_ipinfo.timer [production]
22:15 <mutante> puppetmaster2001 - sudo /usr/local/bin/geoipupdate_job after adding new shell command and timer - succesfully downloaded enterprise database for T288844 [production]