601-650 of 10000 results (19ms)
2020-05-29 §
06:36 <mutante> deneb - systemctl start docker-reporter-releng-images [production]
05:20 <marostegui> Deploy schema change on db1138 (no longer s4 master) - T250055 [production]
05:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1081 to s4 master and remove read-only from s4 T253808', diff saved to https://phabricator.wikimedia.org/P11334 and previous config saved to /var/cache/conftool/dbconfig/20200529-050224-marostegui.json [production]
05:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s4 as read-only for maintenance T253808', diff saved to https://phabricator.wikimedia.org/P11333 and previous config saved to /var/cache/conftool/dbconfig/20200529-050153-marostegui.json [production]
05:00 <marostegui> Starting s4 failover from db1138 to db1081 -T253808 [production]
04:25 <marostegui> Start topology changes in s4 - T253808 [production]
2020-05-28 §
23:48 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.34/skins/Vector/resources/skins.vector.styles/Menu.less: T253912 Hotfix: Cannot rename emptyPortlet to empty-portlet yet (duration: 00m 59s) [production]
23:02 <bd808> `/usr/local/sbin/maintain-dbusers --debug harvest-replicas` (T253930) [admin]
22:41 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.34/extensions/WikibaseMediaInfo/src/Services/FilePageLookup.php: T253792 Follow-up 1827c7a: Ensure inNamespace() is called only on Title object (duration: 00m 58s) [production]
22:24 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T253821 Update MachineVision block list for 2020-05-27 (duration: 00m 57s) [production]
22:09 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Move one CheckUser right change next to the other (duration: 00m 57s) [production]
22:06 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Remove version wrapper around wgOverrideUcfirstCharacters; always true (duration: 00m 59s) [production]
21:48 <jforrester@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.34 [production]
21:26 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.34/includes/filerepo/FileRepo.php: T253922 Mark two FileRepo functions public (duration: 01m 07s) [production]
21:23 <bd808> Admins killed `node dist/index.js` process running on login.toolforge.org bastion; Use a Kubernetes container or the Job Grid please [tools.lziad]
21:19 <bd808> Killed 7 python processes run by user 'mattho69' on login.toolforge.org [tools]
21:12 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.34/includes/specials/SpecialUserrights.php: T253909 Restore visibility (previously implicitely public) (duration: 01m 06s) [production]
21:06 <bstorm_> upgrading tools-k8s-worker-[30-60] to kubernetes 1.16.10 T246122 [tools]
20:38 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.32/skins/Vector/resources/skins.vector.styles: T253905 HOTFIX: Do not apply p-personal absolute positioning to all menus (duration: 01m 07s) [production]
20:22 <shdubsh> restart varnishmtail and atsmtail eqsin [production]
20:11 <shdubsh> restart ncredirmtail on ncredir5001 [production]
19:20 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: roll back the train due to T253905 [production]
19:20 <twentyafterfour> group2 back to wmf.32 due to T253905 [production]
19:20 <milimetric@deploy1001> Finished deploy [analytics/refinery@f6d73c8] (thin): Hotfix #2 today (thin): forgot jars [analytics/refinery@f6d73c8] (duration: 00m 09s) [production]
19:20 <milimetric@deploy1001> Started deploy [analytics/refinery@f6d73c8] (thin): Hotfix #2 today (thin): forgot jars [analytics/refinery@f6d73c8] [production]
19:17 <milimetric@deploy1001> Finished deploy [analytics/refinery@f6d73c8]: Hotfix #2 today: forgot jars [analytics/refinery@f6d73c8] (duration: 16m 54s) [production]
19:14 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.34 refs T253022 [production]
19:01 <shdubsh> restart varnishmtail and atsmtail on cp5001.eqsin.wmnet [production]
19:00 <milimetric@deploy1001> Started deploy [analytics/refinery@f6d73c8]: Hotfix #2 today: forgot jars [analytics/refinery@f6d73c8] [production]
18:31 <milimetric> after deployment, restarted four oozie jobs with new SLAs and fixed datasets definitions [analytics]
17:54 <bstorm_> upgraded tools-k8s-worker-[11..15] and starting on -21-29 now T246122 [tools]
17:52 <James_F> Zuul: [integration/docroot] Switch from php56 to php70 [releng]
17:03 <twentyafterfour@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.34 refs T253022 (duration: 01m 06s) [production]
17:02 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.34 refs T253022 [production]
16:32 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.34/extensions/Wikibase: T253804 Use ThrowingEntityTermStoreWriter when writers shouldn't be called (duration: 01m 15s) [production]
16:10 <James_F> Zuul: Move MediaModeration to production section and enforce T247943 [releng]
16:01 <bstorm_> kubectl upgraded to 1.16.10 on all bastions T246122 [tools]
15:58 <arturo> upgrading tools-k8s-worker-[1..10] to 1.16.10 (T246122) [tools]
15:41 <arturo> upgrading tools-k8s-control-3 to 1.16.10 (T246122) [tools]
15:37 <milimetric@deploy1001> Finished deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182] (duration: 00m 10s) [production]
15:37 <milimetric@deploy1001> Started deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182] [production]
15:17 <arturo> upgrading tools-k8s-control-2 to 1.16.10 (T246122) [tools]
15:09 <arturo> upgrading tools-k8s-control-1 to 1.16.10 (T246122) [tools]
15:05 <milimetric@deploy1001> Finished deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182] (duration: 25m 59s) [production]
15:02 <moritzm> installing exim4 security updates on jessie (stretch/buster already fixed) [production]
14:49 <arturo> cleanup /etc/apt/sources.list.d/ directory in all tools-k8s-* VMs [tools]
14:39 <milimetric@deploy1001> Started deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182] [production]
14:33 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:30 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:26 <ZI_Jony> restarted CVNBot14 [cvn]