|
2019-04-15
ยง
|
| 23:39 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 23:39 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 23:20 |
<cdanis> |
cdanis@icinga1001.wikimedia.org ~ % sudo systemctl restart tcpircbot-logmsgbot.service |
[production] |
| 23:17 |
<bd808> |
scap: SWAT: [[gerrit:497423|wikitech: Use cn:caseExactMatch: as account search filter]] (T165795) |
[production] |
| 20:59 |
<thcipriani> |
gerrit back |
[production] |
| 20:57 |
<gehel> |
shutting down blazegraph and updater on wdqs1010, waiting for data reimport |
[production] |
| 20:55 |
<thcipriani> |
gerrit restart to pick up gc log changes incoming |
[production] |
| 20:37 |
<arlolra> |
Updated Parsoid to 83c17fc9 |
[production] |
| 20:23 |
<Amir1> |
the ores deployment is over |
[production] |
| 19:49 |
<XioNoX> |
export BGP communities (prepend x3 outside asia) to AS3491 in eqsin |
[production] |
| 19:46 |
<mutante> |
bromine/vega: rm /etc/rsyncd.conf ; systemctl stop rsync (clean up old rsync config gerrit:503961) |
[production] |
| 19:45 |
<XioNoX> |
update (and add) AS3491 BGP communities in eqsin |
[production] |
| 18:58 |
<XioNoX> |
update mr1-* security policies - T219384 |
[production] |
| 18:41 |
<onimisionipe> |
depooling maps2003 for psotgres init |
[production] |
| 18:40 |
<onimisionipe> |
pooling map2002 - postgres init complete |
[production] |
| 18:39 |
<Amir1> |
Morning SWAT is done |
[production] |
| 18:35 |
<shdubsh> |
logstash1009: disabling puppet and testing logstash config |
[production] |
| 18:09 |
<mutante> |
LDAP - adding legoktm and qchris to gerritadmin group (T219086) |
[production] |
| 17:45 |
<anomie> |
Backporting fix for T220991 |
[production] |
| 17:41 |
<akosiaris> |
force puppet agent run on maps* after moving config-vars.yaml file for kartotherian, tilerator, tileratorui T220982 |
[production] |
| 17:33 |
<mutante> |
LDAP - re-adding 'pbj' to 'nda' group, extended access until May 6th, transparency report contractor |
[production] |
| 17:23 |
<mutante> |
wikibugs - qdel'ed jobs and restarted another time, make it rejoin |
[production] |
| 17:17 |
<onimisionipe> |
wdqs deployment is complete! for some reasons I don't know scap did not logging here |
[production] |
| 17:17 |
<herron> |
restarted logstash on logstash1007 |
[production] |
| 17:15 |
<mutante> |
restarted wikibugs because it stopped talking |
[production] |
| 16:08 |
<onimisionipe> |
pooling maps2001 - postgres reinit is complete |
[production] |
| 15:55 |
<Reedy> |
changed /srv/mediawiki/docroot/wikimedia.org to a symlink to standard-docroot |
[production] |
| 15:53 |
<XioNoX> |
add cloud-in4 firewall filter to codfw - T211921 |
[production] |
| 15:31 |
<onimisionipe> |
restarting prometheus-wmf-elasticsearch-exporter-9* on all elastic nodes |
[production] |
| 15:30 |
<onimisionipe> |
restarting prometheus-wmf-elasticsearch-exporter-9200 on all elastic nodes |
[production] |
| 15:28 |
<_joe_> |
systemctl reset-failed on ms-be1027, debmonitor session |
[production] |
| 15:24 |
<Amir1> |
end of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T219871) |
[production] |
| 14:55 |
<gehel> |
deploying tilerator to maps1001 to validate deployment is working - T220982 |
[production] |
| 14:55 |
<Amir1> |
start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T219871) |
[production] |
| 14:43 |
<_joe_> |
running apply-config-tilerator on maps1001 |
[production] |
| 14:40 |
<_joe_> |
running apply-config-karthoterian on maps1001 |
[production] |
| 14:22 |
<cdanis> |
T220982 cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'maps1*' 'sudo chmod -R a+r /srv/deployment/tilerator /srv/deployment/kartotherian' |
[production] |
| 14:21 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'maps1*' "disable-puppet 'bad permissions - T220982 - cdanis'" |
[production] |
| 14:18 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'maps*' 'sudo chmod -R a+r /srv/deployment/tilerator /srv/deployment/kartotherian' |
[production] |
| 14:18 |
<gehel> |
reseting permissions on maps server fir /srv/deployment/kartotherian and /srv/deplyoment/tilerator |
[production] |
| 14:04 |
<moritzm> |
rebooting ms-fe1005 for combined kernel/glibc/OpenSSL update |
[production] |
| 13:57 |
<jbond42> |
upgrading puppet 4 -> 5 and facter 2 -> 3 on mediawiki::canary_appserver, mediawiki::appserver::canary_api and cache::cache roles |
[production] |
| 13:56 |
<gehel> |
restart tilerator / kartotherian on all maps servers for openssl update |
[production] |
| 13:55 |
<godog> |
start ms-be1013 decom - T220590 |
[production] |
| 13:42 |
<godog> |
reboot ms-be1013 |
[production] |
| 13:09 |
<moritzm> |
installing wget security updates on trusty hosts |
[production] |
| 12:59 |
<moritzm> |
restarting archiva on archiva1001 for OpenJDK security update |
[production] |
| 12:50 |
<moritzm> |
restarting Apache on matomo1001 to pick up OpenSSL update |
[production] |
| 12:14 |
<moritzm> |
rolling restart of HHVM/Apache on deployment servers to pick up OpenSSL update |
[production] |
| 11:59 |
<fsero> |
pointing boron docker builds to the new registry temporarily (docker builds on boron might fail) |
[production] |