3251-3300 of 10000 results (29ms)
2020-07-13 §
15:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1130', diff saved to https://phabricator.wikimedia.org/P11882 and previous config saved to /var/cache/conftool/dbconfig/20200713-155240-marostegui.json [production]
15:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11881 and previous config saved to /var/cache/conftool/dbconfig/20200713-154847-marostegui.json [production]
15:39 <mholloway-shell@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
15:35 <mholloway-shell@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
15:30 <mholloway-shell@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
14:50 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Stop setting DiscussionToolsEnableVisual, default value (duration: 00m 57s) [production]
14:17 <moritzm> removing lilypond from production T257066 [production]
13:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11880 and previous config saved to /var/cache/conftool/dbconfig/20200713-133604-marostegui.json [production]
13:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1082', diff saved to https://phabricator.wikimedia.org/P11879 and previous config saved to /var/cache/conftool/dbconfig/20200713-133535-marostegui.json [production]
13:05 <kormat@cumin1001> dbctl commit (dc=all): 'Fully repool es1022, and set es1020 to zero weight T257284', diff saved to https://phabricator.wikimedia.org/P11878 and previous config saved to /var/cache/conftool/dbconfig/20200713-130532-kormat.json [production]
12:08 <kormat@cumin1001> dbctl commit (dc=all): 'Start repooling es1022 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11873 and previous config saved to /var/cache/conftool/dbconfig/20200713-120818-kormat.json [production]
11:49 <Urbanecm> Password reset for User:Alert5 (T257806) [production]
11:44 <akosiaris> repool ganeti1007 T244530. Start emptying ganeti1008 [production]
11:08 <Urbanecm> EU B&C done [production]
11:06 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 896c042296b4e1f5d88f786981537655e5d9fea9: Enable SandboxLink extension in trwiki (T256782) (duration: 00m 56s) [production]
10:44 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:612175| Bumping portals to master (612175)]] (duration: 00m 56s) [production]
10:43 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:612175| Bumping portals to master (612175)]] (duration: 00m 56s) [production]
09:42 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:39 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:58 <ema> cp: rolling ats-backend-restart to apply SyslogIdentifier changes -> https://gerrit.wikimedia.org/r/c/operations/puppet/+/611311 [production]
08:57 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T248343 Explicitly set visualeditor-enable to 0 when non-default (duration: 00m 57s) [production]
08:44 <kormat@cumin1001> dbctl commit (dc=all): 'Depool es1022 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11871 and previous config saved to /var/cache/conftool/dbconfig/20200713-084449-kormat.json [production]
08:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1093', diff saved to https://phabricator.wikimedia.org/P11870 and previous config saved to /var/cache/conftool/dbconfig/20200713-083902-marostegui.json [production]
08:34 <kormat@cumin1001> dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1022 T257284', diff saved to https://phabricator.wikimedia.org/P11869 and previous config saved to /var/cache/conftool/dbconfig/20200713-083414-kormat.json [production]
08:20 <kormat> reimaging es1022 T257284 [production]
06:54 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
06:53 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:52 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
06:52 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:51 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) [production]
06:50 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:16 <marostegui> Reverse gerrit password on m2 master - T255715 [production]
06:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11868 and previous config saved to /var/cache/conftool/dbconfig/20200713-060410-marostegui.json [production]
05:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11867 and previous config saved to /var/cache/conftool/dbconfig/20200713-055422-marostegui.json [production]
05:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1093 for upgrade', diff saved to https://phabricator.wikimedia.org/P11866 and previous config saved to /var/cache/conftool/dbconfig/20200713-054840-marostegui.json [production]
05:34 <marostegui> Deploy schema change on s3 codfw master, lag will appear on codfw T253276 [production]
05:30 <marostegui> Stop replication on db1082 for schema change and triggers removal T238966 [production]
05:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P11865 and previous config saved to /var/cache/conftool/dbconfig/20200713-052928-marostegui.json [production]
05:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1119 for innodb compression', diff saved to https://phabricator.wikimedia.org/P11864 and previous config saved to /var/cache/conftool/dbconfig/20200713-051428-marostegui.json [production]
2020-07-11 §
19:16 <qchris> Restarting Gerrit on gerrit1001 to switch to new gerrit.war and zuul plugin [production]
19:16 <qchris@deploy1001> Finished deploy [gerrit/gerrit@a71a0df]: Gerrit to v3.2.2-138-g230805407f and zuul plugin to master-12-ge51d7e8 on gerrit1001 (duration: 00m 07s) [production]
19:15 <qchris@deploy1001> Started deploy [gerrit/gerrit@a71a0df]: Gerrit to v3.2.2-138-g230805407f and zuul plugin to master-12-ge51d7e8 on gerrit1001 [production]
19:08 <qchris> Restarting Gerrit on gerrit2001 to switch to new gerrit.war and zuul plugin [production]
18:55 <qchris@deploy1001> Finished deploy [gerrit/gerrit@a71a0df]: Gerrit to v3.2.2-138-g230805407f and zuul plugin to master-12-ge51d7e8 on gerrit2001 (duration: 00m 10s) [production]
18:55 <qchris@deploy1001> Started deploy [gerrit/gerrit@a71a0df]: Gerrit to v3.2.2-138-g230805407f and zuul plugin to master-12-ge51d7e8 on gerrit2001 [production]
2020-07-10 §
21:52 <ryankemper> Started long-running reindex of Elasticsearch indices in `eqiad`, `codfw`, and `dewiki` on `mwmaint1002` under tmux session `reindex` for user `ryankemper` [production]
20:26 <jgleeson> updated fundraising-tools from 08ba1f6177 to f8e424fe32 [production]
19:02 <mutante> removing firewall hole for gerrit -> mysql servers on dbproxy servers for misc db's [production]
18:44 <mutante> kubernetes1004 - started nagios-nrpe-server [production]
17:57 <ebernhardson> change loginwiki password for Cindy-the-browser-test-bot, no email account was associated to allow for normal reset. [production]