4301-4350 of 10000 results (32ms)
2020-09-16 §
05:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Add es2027 and es2028 to dbctl T261717', diff saved to https://phabricator.wikimedia.org/P12595 and previous config saved to /var/cache/conftool/dbconfig/20200916-053918-marostegui.json [production]
05:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12594 and previous config saved to /var/cache/conftool/dbconfig/20200916-053507-marostegui.json [production]
05:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 into vslow', diff saved to https://phabricator.wikimedia.org/P12593 and previous config saved to /var/cache/conftool/dbconfig/20200916-052343-marostegui.json [production]
05:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12592 and previous config saved to /var/cache/conftool/dbconfig/20200916-052241-marostegui.json [production]
05:07 <marostegui> Repool labsdb1010 [production]
02:22 <mutante> deneb - sudo systemctl start package_builder_Clean_up_build_directory to fix icinga alert after failed build attempts [production]
2020-09-15 §
23:20 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.9/extensions/FlaggedRevs/backend/FlaggedRevsHooks.php: 1c0b0d161fe1024d6d08a27bbacf5b62c56c9c01: Fix APCOND_FR_NEVERBLOCKED handling (T262970) (duration: 00m 56s) [production]
23:18 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.8/extensions/FlaggedRevs/backend/FlaggedRevsHooks.php: 5beace32a396adfcce46b04e7f969b2f9f9effda: Fix APCOND_FR_NEVERBLOCKED handling (T262970) (duration: 00m 58s) [production]
23:14 <urbanecm@deploy1001> Synchronized wmf-config/flaggedrevs.php: ac8bd3894f2dc8f2735cc9fa7b860af1d91c6707: flaggedrevs: Remove non-existent config options (duration: 00m 58s) [production]
23:07 <urbanecm@deploy1001> Scap failed!: Call to mwscript eval.php stderr: not empty [production]
23:00 <urbanecm@deploy1001> Synchronized wmf-config/abusefilter.php: 62b21d55a8f0a94b8cd268d5024df0cf64013dd5: Revert "Remove abusefilter-view right grant from wmf-config" (T255506) (duration: 00m 59s) [production]
20:44 <brennen> removing extraneous recursive symlink /srv/mediawiki-staging/php-1.36.0-wmf.9/php-1.36.0-wmf.8 [production]
18:32 <Urbanecm> Morning B&C done [production]
18:28 <urbanecm@deploy1001> Synchronized wmf-config/abusefilter.php: 084729b7fd0716f11265f1b37570afc120b27109: Remove abusefilter-view right grant from wmf-config (T255506) (duration: 00m 56s) [production]
18:12 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 1d3456570b80b1d8af1d2b71975496e54f87b24e: Enable MediaWiki client errors on frwiki (T255585) (duration: 00m 57s) [production]
18:08 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 79004b7e503c7274fa56d2699b423b6919fbc869: Enable the reverted tag on all wikis (T164307) (duration: 00m 56s) [production]
17:59 <krinkle@deploy1001> Synchronized src/ServiceConfig.php: If727ae4335 (duration: 00m 56s) [production]
17:43 <ppchelko@deploy1001> Finished deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint, feeds time out (duration: 37m 42s) [production]
17:05 <ppchelko@deploy1001> Started deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint, feeds time out [production]
17:05 <ppchelko@deploy1001> Finished deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint (duration: 86m 46s) [production]
17:00 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:59 <hnowlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:57 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
15:38 <ppchelko@deploy1001> Started deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint [production]
15:33 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
15:33 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
15:30 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
15:30 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
15:28 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
15:28 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
15:26 <shdubsh> manual install prometheus-icinga-exporter upgrade on icinga2001 [production]
14:53 <godog> switch grafana to eqiad - T259143 [production]
14:48 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:42 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
14:38 <XioNoX> remove old SNMP community from all network devices [production]
14:23 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
14:22 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: wgEventStreams: Set canary_events_enabled: true for eventlogging_TemplateWizard - T251609 (duration: 00m 56s) [production]
14:21 <otto@deploy1001> sync-file aborted: wgEventStreams: Set canary_events_enabled: true for eventlogging_TemplateWizard - T251609 (duration: 00m 06s) [production]
14:01 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
14:01 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
13:51 <cdanis@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
13:51 <cdanis@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
13:50 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
13:50 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
13:18 <elukey@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
13:14 <cmjohnson1> beginning work inside racks c2, c3, c4 and c5 eqiad [production]
12:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 from vslow, s8, add db1092 temporarily', diff saved to https://phabricator.wikimedia.org/P12589 and previous config saved to /var/cache/conftool/dbconfig/20200915-121849-marostegui.json [production]
12:18 <jbond42> update libxml2 on stretch and jessie [production]
12:08 <jbond42> rolling restart of php7.2-fpm [production]
12:05 <elukey> roll restart cassandra on aqs* to pick up openjdk upgrades [production]