851-900 of 1724 results (10ms)
2022-05-05 §
21:33 <brennen@deploy1002> scap failed: average error rate on 7/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org for details) [production]
2022-04-04 §
12:31 <ottomata> deleting empty typo topics from kafka main-eqiad: eqiad.mediawiki.page-edit (found while working on T241178) [production]
2022-03-31 §
23:45 <mutante> gitlab2001 - fdisk /dev/vdb (g, w) (create partition table), (n, w) (create partition) ; mkfs.ext4 /dev/vdb1 (create filesystem); systemctl reset-failed (fix Icinga alert); mkdir /mnt/gitlab-backup; mount /dev/vdb1 /mnt/gitlab-backup ; blkid (get UUID); edit /etc/fstab and insert "UUID=c5235682-ac21-46a9-85ee-9603f694a6a4 /mnt/gitlab-backup ext4 errors=remount-ro 0 2" T274463 [production]
08:19 <daniel@deploy1002> Synchronized php-1.39.0-wmf.5/extensions/GrowthExperiments/modules/ext.growthExperiments.PostEdit/index.js: Backport: [[gerrit:775370|Post-edit dialog: check for presence of preferences.topicFilters (T305057)]] (duration: 00m 53s) [production]
2022-03-22 §
22:20 <ryankemper> T301511 Mutated cirrus codfw cluster settings to what [I think] they should be, see https://phabricator.wikimedia.org/T301511#7798415; forcing re-check [production]
22:09 <ryankemper> T301511 Forcing recheck of codfw cirrus setting check [production]
21:35 <ryankemper> T301511 Fixed elastic* eqiad cross-cluster search settings (see https://phabricator.wikimedia.org/T301511#7798267) to resolve the `ElasticSearch setting check` alerts in eqiad [production]
10:26 <_joe_> running check-restart-php on api appservers [production]
10:22 <_joe_> running check-and-restart on mw-eqiad-appservers [production]
2022-03-21 §
16:55 <taavi@deploy1002> Synchronized php-1.39.0-wmf.1/extensions/WikimediaEvents/includes/PageSplitter/PageSplitterHooks.php: Backport: [[gerrit:772364|PageSplitter: check for OutputPage::getTitle() returning null (T304331)]] (duration: 00m 50s) [production]
16:53 <taavi@deploy1002> Synchronized php-1.38.0-wmf.26/extensions/WikimediaEvents/includes/PageSplitter/PageSplitterHooks.php: Backport: [[gerrit:772363|PageSplitter: check for OutputPage::getTitle() returning null (T304331)]] (duration: 00m 51s) [production]
2022-03-17 §
16:05 <oblivian@puppetmaster1001> conftool action : edit; selector: name=random_q [production]
2022-03-02 §
21:59 <brennen@deploy1002> Synchronized php-1.38.0-wmf.24/extensions/Linter/includes/Hooks.php: Backport: [[gerrit:767104|Hooks.php: Check for non-array $tags (T302918)]] (duration: 00m 50s) [production]
08:09 <godog> test thanos 0.24.0 on thanos-fe2001 to check if https://github.com/thanos-io/thanos/issues/4531 is fixed [production]
2022-02-23 §
14:11 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.23/extensions/DiscussionTools/includes/Hooks/HookUtils.php: 78f0d9d: Fix check for enabling features on mobile (T302388) (duration: 00m 49s) [production]
14:10 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.22/extensions/DiscussionTools/includes/Hooks/HookUtils.php: 815b3d1: Fix check for enabling features on mobile (T302388) (duration: 00m 50s) [production]
04:36 <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.22/includes/page/ParserOutputAccess.php: Backport: [[gerrit:764838|ParserOutputAccess: Check for latest revision when checking for cache (T283029)]] (duration: 00m 50s) [production]
04:33 <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.23/includes/page/ParserOutputAccess.php: Backport: [[gerrit:764837|ParserOutputAccess: Check for latest revision when checking for cache (T283029)]] (duration: 00m 51s) [production]
2022-02-18 §
17:26 <ariel@deploy1002> Finished deploy [dumps/dumps@f7c16d4]: noop script, dup jobname check for api jobs, do flow dumps in pieces like stubs (duration: 00m 03s) [production]
17:26 <ariel@deploy1002> Started deploy [dumps/dumps@f7c16d4]: noop script, dup jobname check for api jobs, do flow dumps in pieces like stubs [production]
2022-02-15 §
18:50 <bblack> cr[12]-eqiad - edit static fallback for low-traffic (lvs1015 -> lvs1019) [production]
17:08 <bblack> cr[12]-eqiad: manual edit static fallback route for high-traffic2 from lvs1014 to lvs1018 - T301142 [production]
2022-02-10 §
19:11 <bblack> lvs1017 rebooting for sanity-check after prod config - T301142 [production]
2022-02-01 §
00:11 <catrope@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:758033|Lower The Wikipedia Library extension edit count (T288070)]] (duration: 00m 50s) [production]
2022-01-19 §
23:35 <mutante> puppetmaster1001 - revoked puppet cert miscweb.discovery.wmnet; updated kube_services.crts.yaml to include static-bugzilla.wikimedia.org, removed miscweb.discovery.wmnet.crt and .csr.pem, used cergen to check and regenerate cert, committed in private repo, ran puppet on deploy1001 - checked cert in /etc/helmfile-defaults/private/main_services/miscweb/eqiad.yaml with 'openssl x509 [production]
2022-01-18 §
12:36 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.38.0-wmf.17/extensions/GrowthExperiments/modules/ext.growthExperiments.PostEdit/index.js: Backport: [[gerrit:754129|Post-edit dialog: Reload page upon dialog closing for structured tasks (T299188)]] (duration: 00m 51s) [production]
2021-12-20 §
09:56 <marostegui> Stop mysql on db2135 to check new haproxy on bullseye T295965 [production]
09:46 <marostegui> Stop mysql on db2078:3325 to check new haproxy on bullseye T295965 [production]
2021-12-14 §
19:51 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 40f0cff8da7c4484e1fe93b9d649fd03f462e434: VE on zh.wiki: Enable single-edit-tab mode, and other config like en.wiki (T296269) (duration: 00m 57s) [production]
2021-12-13 §
19:40 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.12/extensions/VisualEditor/includes/VisualEditorHooks.php: fa01addb8ae29a97f6e141ec37a5e1204f0d7810: Check VisualEditorDisableForAnons in getEditPageEditor() (T296269) (duration: 00m 56s) [production]
19:39 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/VisualEditor/includes/VisualEditorHooks.php: 8144ab6577bdc93b4d81d5f8541437b746752610: Check VisualEditorDisableForAnons in getEditPageEditor() (T296269) (duration: 00m 56s) [production]
2021-12-10 §
00:23 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:745385|Revert "VE on zh.wiki: Enable single-edit-tab mode" (T296269)]] (duration: 00m 56s) [production]
2021-12-09 §
19:21 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 4e6cba0d3590446bb02815b65ba1c4ae9ed7bfac: VE on zh.wiki: Enable single-edit-tab mode (T296269) (duration: 01m 05s) [production]
2021-12-02 §
01:08 <ryankemper> T280001 Sanity check of `sudo ipvsadm -L -n` on backup `lvs2010` and `lvs1016` looks good (for ex `lvs1016` has `TCP 10.2.2.67:443 wrr`) [production]
2021-11-29 §
20:21 <tgr@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php: Backport: [[gerrit:742260|SuggestedEdits: Drop isActivated() check in getJsData (T296626)]] (duration: 00m 56s) [production]
2021-11-25 §
07:10 <jelto> downtime PyBal backends health check on lvs1015 and lvs1016 for helm3 de-deploy T251305. I'm keeping an eye on icing and remove downtime as soon as I'm finished [production]
2021-11-24 §
10:06 <jelto> downtime PyBal backends health check for helm3 de-deploy T251305. I'm keeping an eye on icing and remove downtime as soon as I'm finished [production]
2021-11-15 §
18:00 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 00650753d77d7a526b6751669bf3548cf81fb02a: foundationwiki: Revoke edit from * (T294900) (duration: 00m 56s) [production]
2021-11-03 §
21:56 <ryankemper> T294961 [WCQS] Forcing recheck of `PyBal IPVS diff check` and `PyBal backends health check` [production]
2021-11-01 §
11:01 <urbanecm> 11:01:21 Synchronized wmf-config/CommonSettings.php: b9aa3d21bfb16aaa9605e7abe311eb122009d6ed: Add edit-legal to editprotected grant (duration: 00m 54s) [production]
2021-10-20 §
23:27 <tstarling@deploy1002> Synchronized wmf-config/InitialiseSettings.php: fawiki require login to edit main namespace T291018 (duration: 01m 04s) [production]
2021-10-19 §
21:34 <mutante> mwmaint1002 - delete large files over 100MB from puppet clientbucket. sudo /usr/bin/find /var/lib/puppet/clientbucket/ -type f -size +100M -delete | fixed Icinga alert: RECOVERY - Check for large files in client bucket on mwmaint1002 is OK: OK: T165885 [production]
2021-10-14 §
16:24 <dancy@deploy1002> Synchronized php-1.38.0-wmf.4/extensions/Collection/includes/CollectionHooks.php: Backport: [[gerrit:730580|Check that the timestamp key/value is set to avoid undefined offset (T293300)]] (duration: 01m 04s) [production]
15:02 <dancy@deploy1002> Synchronized php-1.38.0-wmf.4/extensions/Collection/includes/CollectionHooks.php: Backport: [[gerrit:730580|Check that the timestamp key/value is set to avoid undefined offset (T293300)]] (duration: 01m 03s) [production]
2021-10-07 §
00:11 <mutante> [grafana2001:~] $ sudo systemctl start rsync-var-lib-grafana because of "PROBLEM - Check systemd state on grafana2001 is CRITICAL: CRITICAL - degraded" because of some race condition where a file vanished during sync [production]
2021-09-27 §
18:52 <otto@deploy1002> scap failed: average error rate on 6/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) [production]
2021-09-23 §
22:05 <ryankemper> T280001 [Sanity check] `TCP 10.2.2.67:443 wrr` shows up on `ryankemper@lvs1016:~$ sudo ipvsadm -L -n` and `TCP 10.2.1.67:443 wrr` shows up on `ryankemper@lvs2010:~$ sudo ipvsadm -L -n` as expected [production]
21:00 <ryankemper> T280001 Sanity check of `sudo ipvsadm -L -n` on low-traffic backups `lvs2010` and `lvs1016` looks good, proceeding [production]
21:00 <ryankemper> T280001 Sanity check of `sudo ipvsadm -L -n` on backup `lvs2010` and `lvs1016` looks good, proceeding [production]
2021-09-22 §
18:32 <legoktm@deploy1002> Synchronized php-1.38.0-wmf.1/extensions/GrowthExperiments/modules/help/ext.growthExperiments.PostEditPanel.js: Post-edit Panel: Set task.pageviews to null rather than undefined (T291510) (duration: 01m 05s) [production]