1351-1400 of 10000 results (43ms)
2021-09-20 §
11:18 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:722348|Disable jQuery Migrate on group1 (T280944)]] (duration: 00m 56s) [production]
11:16 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: b518d8ba03e85afdf98f2e06bf569b4f2b551b1b: Mentor dashboard: Enable beta mode at testwiki (T281534) (duration: 00m 55s) [production]
11:11 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.23/extensions/GrowthExperiments/includes/: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 5) (duration: 00m 56s) [production]
11:10 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.23/extensions/GrowthExperiments/ServiceWiring.php: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 4) (duration: 00m 56s) [production]
11:09 <hnowlan> roll restarting restbase service in codfw [production]
11:08 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.23/extensions/GrowthExperiments/includes/MentorDashboard/Modules/MentorTools.php: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 2) (duration: 00m 55s) [production]
11:07 <urbanecm@deploy1002> sync-file aborted: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 1) (duration: 00m 00s) [production]
11:07 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.23/extensions/GrowthExperiments/includes/MentorDashboard/MentorTools/MentorStatusManager.php: b9031bc572f6e3f4e12e6102c2816467af3580f4: Mentor dashboard: Mentor tools (T280307; 1) (duration: 00m 57s) [production]
11:05 <hnowlan> roll restarting restbase service in eqiad for openssl updates [production]
10:45 <hnowlan> roll restarting kartotherian and tilerator on maps2* [production]
10:41 <hnowlan> roll restarting kartotherian and tilerator on maps1* [production]
10:36 <jynus> rolling restart bacula & minio daemons on backup hosts [production]
09:59 <moritzm> restarting apache2 on thorium [production]
09:48 <hnowlan@cumin1001> START - Cookbook sre.postgresql.postgres-init [production]
09:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove s10 from eqiad T167973', diff saved to https://phabricator.wikimedia.org/P17300 and previous config saved to /var/cache/conftool/dbconfig/20210920-094739-marostegui.json [production]
09:10 <moritzm> installing openssl1.0 updates for stretch with backport for forthcoming Let's encrypt issuance chain update (T283165) [production]
08:35 <moritzm> updating clamav on ticket.wikimedia.org/otrs1001 to 0.103.3 [production]
08:02 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:58 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:49 <moritzm> uploaded maps-deduped-tilelist 0.0.3~deb10u1 to buster-wikimedia/main T290982 [production]
07:48 <moritzm> uploaded maps-deduped-tilelist 0.0.3~deb10u1 to buster-wikimedia/main [production]
07:48 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:43 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:35 <marostegui> Stop db1168 and db2129 in sync T167973 [production]
07:34 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:34 <urbanecm@deploy1002> Synchronized wmf-config/throttle.php: af9d6e4e29e5f53ad8cf5aa2c235d54500c433bd: Revert "Add throttle rule for Czech wiki course" (duration: 00m 56s) [production]
07:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1168 T167973', diff saved to https://phabricator.wikimedia.org/P17299 and previous config saved to /var/cache/conftool/dbconfig/20210920-073256-marostegui.json [production]
07:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1096:3316 T167973', diff saved to https://phabricator.wikimedia.org/P17298 and previous config saved to /var/cache/conftool/dbconfig/20210920-073206-marostegui.json [production]
07:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096:3316 T167973', diff saved to https://phabricator.wikimedia.org/P17297 and previous config saved to /var/cache/conftool/dbconfig/20210920-073141-marostegui.json [production]
07:31 <moritzm> uploaded PHP 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf2 to apt.wikimedia.org (component/php7.2 for buster-wikimedia) T291052 [production]
07:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:28 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 8c1d665b5e83f6b1dd1cc4a9c367cb6881473bba: enwiki: Bump Growth features to 25% (mentorship limited to 20% of those users) (T290927) (duration: 00m 57s) [production]
07:20 <urbanecm> Revert undeployed config patch (https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/721959); not even pulled to deployment, so assuming it never hit prod (T289771) [production]
06:00 <marostegui> Upgrade db2071, db2072, db2094 [production]
2021-09-18 §
01:47 <ladsgroup@deploy1002> Synchronized php-1.37.0-wmf.23/includes/libs/rdbms/database/Database.php: (no justification provided) (duration: 00m 57s) [production]
01:01 <ladsgroup@deploy1002> Synchronized php-1.37.0-wmf.23/includes/libs/rdbms/database/Database.php: (no justification provided) (duration: 01m 03s) [production]
2021-09-17 §
21:28 <legoktm@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:19 <legoktm@cumin1001> START - Cookbook sre.dns.netbox [production]
19:00 <hnowlan@cumin1001> END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) [production]
17:02 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephosd1022.eqiad.wmnet with reason: REIMAGE [production]
17:02 <hnowlan@cumin1001> START - Cookbook sre.postgresql.postgres-init [production]
17:00 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1022.eqiad.wmnet with reason: REIMAGE [production]
16:48 <hnowlan@cumin1001> END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) [production]
16:27 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:25 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:11 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:04 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:49 <hnowlan@cumin1001> START - Cookbook sre.postgresql.postgres-init [production]