|
2019-12-04
ยง
|
| 13:52 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:50 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:45 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:45 |
<arturo> |
drop puppet prefix `tools-cron`, deprecated and no longer in use |
[tools] |
| 13:43 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:24 |
<bblack@cumin1001> |
conftool action : set/pooled=no; selector: name=dns[345]001.wikimedia.org |
[production] |
| 13:24 |
<onimisionipe> |
downtimed maps1004 - T239728 |
[production] |
| 13:23 |
<bblack> |
dns[345]001 - starting downtimes/etc for reimage to buster... |
[production] |
| 13:02 |
<arturo> |
drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use |
[toolsbeta] |
| 12:50 |
<arturo> |
drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use |
[toolsbeta] |
| 12:31 |
<filippo@cumin1001> |
conftool action : set/pooled=no; selector: name=ms-fe2007.codfw.wmnet |
[production] |
| 12:29 |
<Urbanecm> |
EU SWAT done |
[production] |
| 12:28 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/WikimediaMessages/: SWAT: bbf2a33: Change Schema Revision of WMDEBannerEvents (T239430) (duration: 01m 02s) |
[production] |
| 12:26 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/WikimediaMessages/: SWAT: b3ef5cd: Change Schema Revision of WMDEBannerEvents (T239430) (duration: 01m 04s) |
[production] |
| 11:38 |
<jbond42> |
puppet enabled accross the fleet and new CA certificate installed |
[production] |
| 11:36 |
<elukey> |
restart mariadb on analytics1030 (hadoop test coordinator) to test explicit_defaults_for_timestamp - T236180 |
[analytics] |
| 11:31 |
<akosiaris> |
drain kubernetes1002 for test of nf_conntrack changes |
[production] |
| 11:23 |
<jbond42> |
enable puppet in eqiad and deploy updated CA |
[production] |
| 11:13 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) |
[production] |
| 10:54 |
<jbond42> |
enable puppet in codfw and deploy updated CA |
[production] |
| 10:46 |
<jbond42> |
enable puppet in esams and deploy updated CA |
[production] |
| 10:42 |
<jbond42> |
enable puppet in ulsfo and deploy updated CA |
[production] |
| 10:31 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.restart |
[production] |
| 10:31 |
<gehel> |
rolling restart of wdqs for config change (event logging) - T101013 |
[production] |
| 10:30 |
<jbond42> |
enable puppet in eqsin and deploy updated CA |
[production] |
| 10:24 |
<marostegui> |
stop replication and mysql on db2107 (s2 codfw master) to test puppet CA changes |
[production] |
| 10:21 |
<marostegui> |
stop replication and mysql on db2071 to test puppet CA changes |
[production] |
| 10:02 |
<jbond42> |
disabling puppet accros the fleet to start CA update change 548241 |
[production] |
| 09:29 |
<godog> |
roll-restart logstash7 in codfw/eqiad after https://gerrit.wikimedia.org/r/c/operations/puppet/+/554472 |
[production] |
| 09:15 |
<marostegui> |
Reload labsdb1010 after reimporting wikidatawiki.page - T238399 |
[production] |
| 09:06 |
<moritzm> |
updated jenkins on apt.wikimedia.org to 2.190.3 (T239586) |
[production] |
| 08:05 |
<effie> |
Restart php7-fpm on mw1348 |
[production] |
| 07:09 |
<marostegui> |
Depool labsdb1010 to reimport wikidatawiki.page - T238399 |
[production] |
| 07:02 |
<marostegui> |
Repool labsdb1011 |
[production] |
| 06:36 |
<mutante> |
removed LVS IP for git-ssh from interface on phab1003 |
[production] |
| 06:25 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=phab1001-vcs.eqiad.wmnet |
[production] |
| 06:13 |
<mutante> |
phab1001 - running rsync of /srv/repos with --delete because it's larger than the source by about 5GB - deleting objects to match phab1003, former prod server. now both 50G (T238956) |
[production] |
| 06:04 |
<marostegui> |
Depool labsdb1011 |
[production] |
| 06:01 |
<mutante> |
rsyncing /srv/repos data once again. pulling from phab1003 to phab1001 (T238956) |
[production] |
| 05:51 |
<marostegui> |
Deploy schema change on s3 primary master (db1123) |
[production] |
| 04:59 |
<mutante> |
removed downtime for phabricator.wikimedia.org meta service (paging) |
[production] |
| 04:58 |
<mutante> |
phabricator maintenance ended for today - now running on phab1001 (buster) |
[production] |
| 04:58 |
<mutante> |
install1002 - restarted isc-dhcpd |
[production] |
| 04:39 |
<mutante> |
phab1001 - rebooting for BIOS config change |
[production] |
| 03:26 |
<andrewbogott> |
shutting down VMs as the project is unclaimed https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge |
[wpx] |
| 03:23 |
<andrewbogott> |
shutting down VMs as the project is unclaimed https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge |
[test-twemproxy] |
| 03:20 |
<andrewbogott> |
shutting down VMs as the project is unclaimed https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge |
[social-tools] |
| 03:17 |
<andrewbogott> |
deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge#orig |
[orig] |
| 03:16 |
<andrewbogott> |
shutting down VMs as the project is unclaimed https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge |
[mcr-dev] |
| 03:10 |
<andrewbogott> |
shutting down VMs as the project is unclaimed https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge |
[hat-imagescalers] |