|
2018-09-08
§
|
| 20:22 |
<legoktm> |
killed https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/1503/ because it was taking 2+ hours |
[releng] |
| 20:20 |
<legoktm> |
deployed https://gerrit.wikimedia.org/r/457077 |
[releng] |
| 20:19 |
<legoktm> |
deleted mwext-PoolCounter-* jobs since they're now unused |
[releng] |
| 10:35 |
<gtirloni> |
restarted cron and truncated /var/log/exim4/paniclog (T196137) |
[tools] |
| 09:45 |
<gtirloni> |
tools restarted cron and truncated /var/log/exim4/paniclog (T196137) |
[production] |
| 04:22 |
<krinkle@deploy1001> |
Synchronized multiversion/: Ia27a8f7ed612f (duration: 00m 49s) |
[production] |
| 04:16 |
<krinkle@deploy1001> |
Synchronized wmf-config/profiler.php: Ia27a8f7ed612f (duration: 00m 54s) |
[production] |
| 03:06 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/458990 https://gerrit.wikimedia.org/r/458996 |
[releng] |
| 01:51 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/458956 https://gerrit.wikimedia.org/r/458957 |
[releng] |
| 01:10 |
<mutante> |
also rsyncing /var/lib/tor-instances/ data for second instance and restarting service (T196701) |
[production] |
| 00:53 |
<mutante> |
radium - stopping rsync.service |
[production] |
| 00:27 |
<mutante> |
torrelay1001 - reset internal state (sighup) with "arm" and pressing x twice |
[production] |
| 00:18 |
<legoktm> |
deployed https://gerrit.wikimedia.org/r/458882 |
[releng] |
| 00:18 |
<mutante> |
to watch what is happenin on torrelay1001 - sudo -u debian-tor arm - if asked for password it's in passwords::tor in private |
[production] |
| 00:16 |
<mutante> |
tor relay switched over from radium to torrelay1001, fixed /var/lib/tor permissions, restarted service, flipped DNS CNAME (5M TTL), traffic can be seen with "arm", monitoring all green (T196701) |
[production] |
|
2018-09-07
§
|
| 23:26 |
<mutante> |
ms-be2042 - repairing /dev/sdj1 (T199198) |
[production] |
| 23:25 |
<mutante> |
ms-be2041 - repairing /dev/sdh1 (T199198) |
[production] |
| 23:23 |
<mutante> |
ms-be1041 - repairing xfs per https://wikitech.wikimedia.org/wiki/Swift/How_To#Repair_xfs_free_blocks_counter_corruption (T199198) |
[production] |
| 22:17 |
<mutante> |
gerrit - restarting for config change to move log files to /var/log/gerrit/ |
[production] |
| 22:16 |
<mutante> |
- cobalt (gerrit) - applying change to move log file location, manually moved logs to /var/log/gerrit, remove old log dir, let puppet re-create it, like on gerrit2001 |
[production] |
| 21:31 |
<mutante> |
gerrit2001, moving gerrit logfiles to /var/log/gerrit, removing old gerrit logdir, letting puppet re-create it as symlink |
[production] |
| 21:06 |
<zhuyifei1999_> |
reverted hotpatch, deployed till 3375dc3 |
[quarry] |
| 20:47 |
<zhuyifei1999_> |
hotpatch /etc/uwsgi/apps-enabled/quarry-web.ini processes 8 -> 1 for some gdb-ing |
[quarry] |
| 20:01 |
<marxarelli> |
bringing integration-slave-docker-1006 online again since disk space has been reclaimed |
[releng] |
| 19:56 |
<framawiki> |
deployed 501695f to quarry-main-01 (T202588) |
[quarry] |
| 19:15 |
<Krinkle> |
marked integration-slave-docker-1025 offline (no space), aborted builds manualy |
[releng] |
| 18:30 |
<legoktm> |
started gear_client on contint1001 |
[releng] |
| 18:20 |
<mutante> |
LDAP: correction, 'monipe' replaced with 'onimisionipe' in wmf group (T202708) |
[production] |
| 18:19 |
<marxarelli> |
setting integration-slave-docker-1026 executors to 4 to avoid disk space exhaustion due to concurrent builds |
[releng] |
| 18:18 |
<joal> |
Manually downoad namespaces for 2018-08 |
[analytics] |
| 18:12 |
<mutante> |
LDAP: added user 'monipe' to group 'wmf' (T202708) |
[production] |
| 18:12 |
<Krinkle> |
Marking integration-slave-docker-1026 offline (ENOSPC) |
[releng] |
| 18:11 |
<framawiki> |
deployed 769cace to quarry-main-01 (T202588) |
[quarry] |
| 18:02 |
<legoktm@deploy1001> |
Synchronized php-1.32.0-wmf.20/extensions/EUCopyrightCampaign/: Update MEPs - https://gerrit.wikimedia.org/r/458628 (for real this time) (duration: 00m 50s) |
[production] |
| 17:52 |
<legoktm@deploy1001> |
Synchronized php-1.32.0-wmf.20/extensions/EUCopyrightCampaign/: Update MEPs - https://gerrit.wikimedia.org/r/458628 (duration: 00m 50s) |
[production] |
| 17:45 |
<XioNoX> |
apply firewall changes on pfw3-eqiad - T203793 |
[production] |
| 17:40 |
<XioNoX> |
apply firewall changes on pfw3-codfw - T203793 |
[production] |
| 17:32 |
<joal> |
Manually rerun download-project-namespace-map on analytics1003 after cron's failure |
[analytics] |
| 16:42 |
<XioNoX> |
explicitely permit install1002/2002:80 in filter labs-in4 on cr1/2-eqiad - T190424 |
[production] |
| 14:56 |
<moritzm> |
uploaded linux-meta 1.20+deb9u1 to apt.wikimedia.org/stretch-wikimedia (provides a new meta package for Linux 4.14) |
[production] |