401-450 of 10000 results (68ms)
2020-01-17 §
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P10190 and previous config saved to /var/cache/conftool/dbconfig/20200117-062838-marostegui.json [production]
06:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P10189 and previous config saved to /var/cache/conftool/dbconfig/20200117-061602-marostegui.json [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1081', diff saved to https://phabricator.wikimedia.org/P10188 and previous config saved to /var/cache/conftool/dbconfig/20200117-060259-marostegui.json [production]
02:45 <urandom> bootstrapping restbase2022-b — T243000 [production]
00:45 <mutante> urldownloaders - rm /etc/logrotate.d/squid3 ; systemctl start logrotate (this fixes failed logrotate because of squid3 vs squid file = duplicate entry, but puppet will recreate it) [production]
00:33 <urandom> bootstrapping restbase2022-a — T243000 [production]
2020-01-16 §
22:38 <mutante> ganeti1003 - deleting VM gerrit-test (T239151) [production]
22:37 <dzahn@cumin1001> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) [production]
22:37 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
22:34 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) [production]
22:34 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
22:21 <urandom> bootstrapping restbase2021-c — T243000 [production]
20:40 <mforns@deploy1001> Finished deploy [analytics/refinery@26a587a] (thin): deploying analytics-refinery to accompany refinery-source v0.0.112 (duration: 00m 07s) [production]
20:40 <mforns@deploy1001> Started deploy [analytics/refinery@26a587a] (thin): deploying analytics-refinery to accompany refinery-source v0.0.112 [production]
20:37 <mforns@deploy1001> Finished deploy [analytics/refinery@26a587a]: deploying analytics-refinery to accompany refinery-source v0.0.112 (duration: 14m 06s) [production]
20:29 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.15/extensions/CentralAuth/includes/GlobalRename/GlobalRenameBlacklist.php: Special:GlobalRenameRequest: Initialize blacklist even if empty T242974 (duration: 00m 57s) [production]
20:23 <mforns@deploy1001> Started deploy [analytics/refinery@26a587a]: deploying analytics-refinery to accompany refinery-source v0.0.112 [production]
20:13 <urandom> bootstrapping restbase2021-b — T243000 [production]
20:01 <Urbanecm> Purge 12 logos URLs (T150618) [production]
20:00 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 5a32bde: Add logos to IS.php (T150618) (duration: 00m 56s) [production]
19:58 <urbanecm@deploy1001> Synchronized static/images/project-logos/: SWAT: b558eea: Fix mistakes in HD logos (T150618) (duration: 00m 56s) [production]
19:33 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Enable topic search, behind a hidden preference (T242698) (duration: 00m 56s) [production]
19:15 <arlolra@deploy1001> Finished deploy [parsoid/deploy@7bf9819]: (no justification provided) (duration: 07m 13s) [production]
19:08 <arlolra@deploy1001> Started deploy [parsoid/deploy@7bf9819]: (no justification provided) [production]
19:08 <catrope@deploy1001> Synchronized wmf-config/CommonSettings.php: Remove kask-echoseen-transition definition, now unused (T234963) (duration: 01m 35s) [production]
19:05 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Echo: switch entirely to Kask, remove Redis fallback (T234963) (duration: 00m 56s) [production]
19:02 <arlolra@deploy1001> Finished deploy [parsoid/deploy@7bf9819]: Updating Parsoid to 02f0066 (duration: 08m 30s) [production]
18:54 <arlolra@deploy1001> Started deploy [parsoid/deploy@7bf9819]: Updating Parsoid to 02f0066 [production]
18:01 <urandom> bootstrapping restbase2021-a — T243000 [production]
17:48 <James_F> Manually purged the trwiki logos from Varnish as part of updating them to reflect unblocking, T242977 [production]
17:48 <jforrester@deploy1001> Synchronized static/images/project-logos/trwiki-2x.png: [trwiki] Change logo to reflect unblocking, 2x T242977 (duration: 00m 56s) [production]
17:47 <jforrester@deploy1001> Synchronized static/images/project-logos/trwiki-1.5x.png: [trwiki] Change logo to reflect unblocking, 1.5x T242977 (duration: 00m 55s) [production]
17:46 <jforrester@deploy1001> Synchronized static/images/project-logos/trwiki.png: [trwiki] Change logo to reflect unblocking, 1x T242977 (duration: 00m 56s) [production]
17:39 <effie> Updgrade parsoid to to php 7.2.26 and restart - T241222 [production]
17:05 <dcausse> restarting blazegraph@wdqs1007 (T242453) [production]
17:02 <jakob@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'termbox' for release 'production' . [production]
16:53 <jakob@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'termbox' for release 'production' . [production]
16:51 <jakob@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . [production]
16:31 <dcausse> depooling wdqs1007, blazegraph stuck (T242453) [production]
16:30 <jakob@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'test' . [production]
15:59 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:59 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:59 <effie> Updgrade appservers and api to php 7.2.26 and restart - T241222 [production]
15:16 <elukey@deploy1001> Finished deploy [analytics/superset/deploy@16a1644]: Upgrade to superset 0.35.2 (duration: 00m 40s) [production]
15:15 <elukey@deploy1001> Started deploy [analytics/superset/deploy@16a1644]: Upgrade to superset 0.35.2 [production]
15:04 <vgutierrez> rolling restart of ats-tls. This effectively disables TLSv1/1.1 across the caching cluster - T238038 [production]
14:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1097:3314 db1097:3315', diff saved to https://phabricator.wikimedia.org/P10182 and previous config saved to /var/cache/conftool/dbconfig/20200116-142800-marostegui.json [production]
14:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1097:3314 db1097:3315', diff saved to https://phabricator.wikimedia.org/P10181 and previous config saved to /var/cache/conftool/dbconfig/20200116-140501-marostegui.json [production]
14:04 <liw@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.15 [production]
13:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1097:3314 db1097:3315', diff saved to https://phabricator.wikimedia.org/P10180 and previous config saved to /var/cache/conftool/dbconfig/20200116-135659-marostegui.json [production]