1451-1500 of 10000 results (25ms)
2021-07-02 §
08:24 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw[1420-1421].eqiad.wmnet with reason: setup new appservers in eqiad A3 https://phabricator.wikimedia.org/T279309 [production]
08:24 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: setup new appservers in eqiad A3 https://phabricator.wikimedia.org/T279309 [production]
08:24 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 6 hosts with reason: setup new appservers in eqiad A3 https://phabricator.wikimedia.org/T279309 [production]
08:03 <moritzm> installing ipmitool security updates [production]
07:54 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1268.eqiad.wmnet [production]
07:54 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1267.eqiad.wmnet [production]
07:53 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1268.eqiad.wmnet [production]
07:53 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1267.eqiad.wmnet [production]
07:25 <dcausse> installing openjdk-8-dbg on wdqs1013 [production]
03:14 <ryankemper> T264053 `sudo -E cumin 'P:elasticsearch::cirrus' 'sudo run-puppet-agent --force'` [production]
03:11 <ryankemper> T264053 `sudo -E cumin 'P:elasticsearch::cirrus' 'sudo apt update'` fixed the issue [production]
03:07 <ryankemper> T264053 `Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-madvise' returned 100: Reading package lists...` grr [production]
03:07 <ryankemper> T264053 `ryankemper@elastic2054:~$ sudo run-puppet-agent --force` [production]
03:06 <ryankemper> T264053 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/702791; will run puppet on single host [production]
03:05 <ryankemper> T264053 `sudo -E cumin 'P:elasticsearch::cirrus' 'sudo disable-puppet "verify new deb package works - T264053"'` [production]
03:02 <legoktm> uploaded elasticsearch-madvise_0.1~deb9u1_amd64.changes to stretch-wikimedia on apt1001 [production]
01:47 <eileen> civicrm revision changed from e07c2be1a7 to bb62188ec6, config revision is 1739c53fcb [production]
01:16 <legoktm> uploaded elasticsearch-madvise 0.1 to apt.wm.o (T264053) [production]
2021-07-01 §
23:29 <thcipriani@deploy1002> Synchronized README: Config: [[gerrit:702777|Revert "deployment training: readme whitespace"]] (duration: 00m 56s) [production]
23:21 <thcipriani@deploy1002> Synchronized README: Config: [[gerrit:702774|deployment training: readme whitespace]] (duration: 00m 57s) [production]
22:37 <urbanecm> Start server-side upload for 1 video file (T285182) [production]
22:36 <urbanecm> Start server-side upload for 1 video file (T285789) [production]
22:31 <dancy@deploy1002> Synchronized .pipeline: Config: [[gerrit:702704|Use train-versions.json to map from version to image tag (T282824)]] (duration: 00m 57s) [production]
22:27 <urbanecm> Start server-side upload for 1 video file (T285682) [production]
21:43 <dancy@deploy1002> Synchronized .pipeline/config.yaml: Config: [[gerrit:702755|Temporarily disable notification for security patch failures]] (duration: 00m 57s) [production]
19:45 <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.37.0-wmf.12 [production]
19:41 <brennen@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.12 (duration: 01m 12s) [production]
19:39 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.12 [production]
19:35 <brennen@deploy1002> Synchronized php-1.37.0-wmf.12/tests/phpunit/includes/TitleMethodsTest.php: Backport: [[gerrit:702711|Consistently normalize Title::mFragment before setting (T285951)]] (duration: 01m 10s) [production]
19:34 <brennen@deploy1002> Synchronized php-1.37.0-wmf.12/includes/Title.php: Backport: [[gerrit:702711|Consistently normalize Title::mFragment before setting (T285951)]] (duration: 01m 10s) [production]
19:18 <brennen@deploy1002> Synchronized php-1.37.0-wmf.12/.pipeline/config.yaml: Backport: [[gerrit:702168|Trigger update-train-versions job at end of wmf-publish pipeline]] (duration: 01m 08s) [production]
18:55 <otto@deploy1002> Finished deploy [analytics/refinery@7dea883] (hadoop-test): Deploying to analytics-test cluster for testing gobblin [analytics/refinery@7dea883] (duration: 05m 19s) [production]
18:50 <otto@deploy1002> Started deploy [analytics/refinery@7dea883] (hadoop-test): Deploying to analytics-test cluster for testing gobblin [analytics/refinery@7dea883] [production]
18:43 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 7995f7abe3b94eb0326064cbbd1d3111f8f21365: Use Vue.js for QuickSurveys on available wikis (T285890) (duration: 01m 09s) [production]
18:20 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.12/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php: 654877f92fa18ae766d693630025c69400cad3ac: EventDispatcher: Ensure we fetch page content from the primary database (T285895) (duration: 01m 12s) [production]
18:18 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php: 6d9043087ec421e1321cd6797934928e2651b1c1: EventDispatcher: Ensure we fetch page content from the primary database (T285895) (duration: 01m 14s) [production]
16:33 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:28 <brennen@deploy1002> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.37.0-wmf.12" [production]
16:27 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
16:23 <reedy@deploy1002> Synchronized php-1.37.0-wmf.12/extensions/ConfirmEdit/SimpleCaptcha/SimpleCaptcha.php: T285959 (duration: 01m 20s) [production]
16:11 <vgutierrez> restart varnish-fe on cp3059 - T285953 [production]
14:58 <papaul> poweroff mw2380 for disk replacement [production]
14:57 <jiji@cumin1001> conftool action : set/pooled=inactive; selector: name=mw2380.codfw.wmnet [production]
14:53 <effie> depool mw2380 for disk repair - T285603 [production]
14:51 <jiji@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
14:51 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:45 <moritzm> installing glib2.0 security updates on buster [production]
13:50 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts maps2002.codfw.wmnet [production]
13:35 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts maps2002.codfw.wmnet [production]
13:02 <marostegui> Deploy schema change on s2 eqiad master T276150 [production]