2101-2150 of 10000 results (30ms)
2020-07-14 ยง
08:30 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:30 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:30 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:30 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:30 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:30 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:13 <jforrester@deploy1001> Started scap: Re-re-start full scap to push out wmf.41 and switch testwikis to it T256669 [production]
08:05 <akosiaris> restart pybal on lvs2009 [production]
08:03 <_joe_> restart pybal on lvs1016 [production]
08:02 <akosiaris> restart pybal on lvs2007 [production]
08:01 <akosiaris@cumin1001> conftool action : set/pooled=inactive; selector: name=restbase2009.codfw.wmnet [production]
08:00 <_joe_> restart pybal on lvs1015 [production]
08:00 <akosiaris> restart pybal on lvs2010 after merging https://gerrit.wikimedia.org/r/612487 [production]
07:52 <jforrester@deploy1001> sync aborted: Re-start full scap to push out wmf.41 and switch testwikis to it T256669 (duration: 02m 14s) [production]
07:50 <jforrester@deploy1001> Started scap: Re-start full scap to push out wmf.41 and switch testwikis to it T256669 [production]
07:48 <oblivian@deploy1001> Synchronized wmf-config/InitialiseSettings.php: revert forcehttps in an attempt to fix T257887 (duration: 01m 06s) [production]
07:32 <oblivian@deploy1001> sync-file aborted: revert forcehttps in an attempt to fix T257887 (duration: 00m 20s) [production]
07:31 <oblivian@deploy1001> Scap failed!: 7/9 canaries failed their endpoint checks(http://en.wikipedia.org) [production]
07:27 <moritzm> installing libtasn1-6 security updates [production]
07:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1075', diff saved to https://phabricator.wikimedia.org/P11894 and previous config saved to /var/cache/conftool/dbconfig/20200714-071233-marostegui.json [production]
07:04 <marostegui> Drop gerrit, gerritro, gerrittest users from m2 databases - T255715 [production]
06:58 <marostegui> Stop mysql on db1131 for HW maintenance [production]
06:56 <oblivian@deploy2001> helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
06:54 <jforrester@deploy1001> scap failed: RuntimeError Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 24m 59s) [production]
06:54 <jforrester@deploy1001> Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) [production]
06:53 <oblivian@deploy2001> helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
06:53 <marostegui> Deploy MCR schema change on s5 primary master T238966 [production]
06:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1078', diff saved to https://phabricator.wikimedia.org/P11893 and previous config saved to /var/cache/conftool/dbconfig/20200714-065229-marostegui.json [production]
06:29 <jforrester@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.41 [production]
05:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Decrease a bit db1088 load', diff saved to https://phabricator.wikimedia.org/P11891 and previous config saved to /var/cache/conftool/dbconfig/20200714-051551-marostegui.json [production]
05:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1131 for HW maintenance', diff saved to https://phabricator.wikimedia.org/P11890 and previous config saved to /var/cache/conftool/dbconfig/20200714-050931-marostegui.json [production]
05:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1093 from api', diff saved to https://phabricator.wikimedia.org/P11889 and previous config saved to /var/cache/conftool/dbconfig/20200714-050912-marostegui.json [production]
05:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1093 to s6 master and remove read-only from s6 T257253', diff saved to https://phabricator.wikimedia.org/P11888 and previous config saved to /var/cache/conftool/dbconfig/20200714-050157-marostegui.json [production]
05:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s6 as read-only for maintenance T257253', diff saved to https://phabricator.wikimedia.org/P11887 and previous config saved to /var/cache/conftool/dbconfig/20200714-050039-marostegui.json [production]
05:00 <marostegui> Starting s6 failover from db1131 to db1093 - T257253 [production]
04:59 <James_F> 1.35.0-wmf.41 branched at 7d04152db4f8ea9a459511bed8117101d9bb4602 [production]
04:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1078', diff saved to https://phabricator.wikimedia.org/P11886 and previous config saved to /var/cache/conftool/dbconfig/20200714-043907-marostegui.json [production]
04:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1093 in preparation for failover', diff saved to https://phabricator.wikimedia.org/P11885 and previous config saved to /var/cache/conftool/dbconfig/20200714-041548-marostegui.json [production]
04:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P11884 and previous config saved to /var/cache/conftool/dbconfig/20200714-041440-marostegui.json [production]
01:23 <ryankemper> Started long-running Elasticsearch reindex of `eqiad`, `codfw`, and `cloudelastic`. tmux session `reindex` under `ryankemper` on `mwmaint1002` [production]
01:20 <cdanis> โŒcdanis@lvs1015.eqiad.wmnet ~ ๐Ÿ•ค๐Ÿบ sudo systemctl restart pybal.service [production]
01:15 <cdanis> โœ”๏ธ cdanis@lvs1016.eqiad.wmnet ~ ๐Ÿ•˜๐Ÿบ sudo systemctl restart pybal.service [production]
01:14 <cdanis> โœ”๏ธ cdanis@lvs2009.codfw.wmnet ~ ๐Ÿ•˜๐Ÿบ sudo systemctl restart pybal.service [production]
01:01 <cdanis> โœ”๏ธ cdanis@lvs2010.codfw.wmnet ~ ๐Ÿ•˜๐Ÿบ sudo systemctl restart pybal.service [production]
2020-07-13 ยง
23:06 <mutante> releases* delete /usr/local/sbin/sync-* scripts created by rsync::quickdatacopy and let puppet recreate the ones still needed [production]
22:27 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: I80ca62643f5c (duration: 00m 58s) [production]
20:12 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@1edde21]: airflow: ship_to_es: Implement multi-index understanding (duration: 00m 29s) [production]
20:12 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@1edde21]: airflow: ship_to_es: Implement multi-index understanding [production]
20:03 <mutante> rsynced reprepro data from releases1001 to releases1002, releases2002 [production]
19:50 <eileen> disable target smart job process-control config revision is b00e7680ca [production]