|
2017-03-29
§
|
| 10:12 |
<ema> |
emptying /srv/log/parsoid/main.log.1 (3.2G!) on ruthenium to reclaim some disk space |
[production] |
| 10:11 |
<elukey> |
upgrading ssl cert api.svc.codfw.wmnet to include the new discovery endpoints |
[production] |
| 09:39 |
<ema> |
upgrading twisted to 16.2.0 on lvs200[123] (codfw primaries) T160433 |
[production] |
| 08:54 |
<ema> |
upgrading twisted to 16.2.0 on lvs200[456] (codfw secondaries) T160433 |
[production] |
| 08:39 |
<ema> |
apt.w.o: set digest-algo to sha256 in gpg.conf T132325 |
[production] |
| 08:29 |
<elukey> |
upgrading ssl cert rendering.svc.codfw.wmnet to include the new discovery endpoints |
[production] |
| 07:57 |
<marostegui> |
Convert s6 UNIQUE keys into PK on db1093 - T17441 |
[production] |
| 07:55 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1093 - T17441 (duration: 00m 54s) |
[production] |
| 06:01 |
<marostegui> |
Keep converting UNIQUE keys to PK on s4 - db1091 - T17441 |
[production] |
| 03:15 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Mar 29 03:15:16 UTC 2017 (duration 5m 53s) |
[production] |
| 03:09 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.18) (duration: 14m 55s) |
[production] |
| 02:35 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.17) (duration: 13m 41s) |
[production] |
| 01:41 |
<krinkle@tin> |
Synchronized errorpages/404.php: Match 404.html and default.html - Id58e25afbe (duration: 00m 44s) |
[production] |
| 01:16 |
<mutante> |
rsyncing librenms/torrus/smokeping app data from netmon1001 to gerrit2001. adding alias "syncit" to do it all at once (T125020) |
[production] |
| 00:57 |
<paravoid> |
Removing upload.wikimedia.org/index.html ("swift delete root index.html") from both eqiad/codfw |
[production] |
|
2017-03-28
§
|
| 23:22 |
<thcipriani@tin> |
Synchronized php-1.29.0-wmf.17/extensions/NavigationTiming/modules/ext.navigationTiming.js: SWAT: [[gerrit:345078|ext.NavigationTiming: Restore unsampled Save Timing]] T161368 (duration: 00m 45s) |
[production] |
| 23:16 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:345184|Enable header version 2 on all wikis]] T160471 (duration: 00m 45s) |
[production] |
| 22:45 |
<urandom> |
T111113: Restarting Cassandra instances, eqiad row 'd' {{done]} |
[production] |
| 22:21 |
<mutante> |
DNS - creating new language "dty" (T161529) - running "authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsd checkconf && gdnsd reload-zones" to trigger re-creation of zone files after change in langs.tmpl. (gerrit:345077) | https://www.ethnologue.com/language/dty |
[production] |
| 22:19 |
<mutante> |
DNS - creating new language "dty" (T160865) - running "authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsd checkconf && gdnsd reload-zones" to trigger re-creation of zone files after change in langs.tmpl. (gerrit:345077) | https://www.ethnologue.com/language/dty |
[production] |
| 21:55 |
<urandom> |
T111113: Restarting Cassandra instances, eqiad row 'd' |
[production] |
| 21:55 |
<urandom> |
T111113: Restarting Cassandra instances, eqiad row 'b' {{done}} |
[production] |
| 21:18 |
<andrewbogott> |
upgraded nova-compute on labvirt1014 because it contains a long-awaited bugfix |
[production] |
| 21:08 |
<urandom> |
T111113: Restarting Cassandra instances, eqiad row 'b' |
[production] |
| 21:08 |
<urandom> |
T111113: Restarting Cassandra instances, eqiad row 'a' {{done}} |
[production] |
| 20:24 |
<mutante> |
ms-fe1001 thru msfe1004 - scheduled last downtime for host and services in icinga - shutdown -h now, turn them off, revoke puppet certs, salt-keys... (T160986) |
[production] |
| 20:22 |
<mutante> |
mc1019 - puppet fail due to Failed resource /etc/redis/replica since 4 days |
[production] |
| 20:21 |
<urandom> |
T111113: Restarting Cassandra instances, eqiad row 'a' |
[production] |
| 20:21 |
<mutante> |
copper - puppet errors due to Failed resource /var/lib/docker/devicemapper ?? |
[production] |
| 20:19 |
<mutante> |
mwdebug1002 - same, was low on disk space, 'apt-get clean' freed > 3GB |
[production] |
| 20:18 |
<mutante> |
mwdebug1001 - was low on disk space, 'apt-get clean' - freed about 4GB |
[production] |
| 20:15 |
<mutante> |
mw1261 - depooled |
[production] |
| 20:14 |
<mutante> |
mw1261 runs with HHVM 3.18 - which seems to have a bug leading to a deadlock every 4-5 hours |
[production] |
| 20:14 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.29.0-wmf.18 |
[production] |
| 20:13 |
<mutante> |
mw1261 HHVM crash as predicted by Moritz - ran sudo hhvm-dump-debug. Backtrace saved as /tmp/hhvm.79460.bt. |
[production] |
| 20:06 |
<mutante> |
ms-fe100[1-4] - disable/stop puppet, stop salt minion, decom (T160986) |
[production] |
| 19:57 |
<thcipriani@tin> |
Finished scap: testwiki to php-1.29.0-wmf.18 and rebuild l10n cache (duration: 40m 19s) |
[production] |
| 19:37 |
<mobrovac> |
restbase deploying d477f495 |
[production] |
| 19:33 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'd' {{done}} |
[production] |
| 19:17 |
<thcipriani@tin> |
Started scap: testwiki to php-1.29.0-wmf.18 and rebuild l10n cache |
[production] |
| 18:45 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'd' |
[production] |
| 18:44 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'c' {{done}} |
[production] |
| 18:18 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@1689d86]: Rename event field in logs (duration: 00m 52s) |
[production] |
| 18:18 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@1689d86]: Rename event field in logs |
[production] |
| 17:53 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'c' |
[production] |
| 17:22 |
<thcipriani> |
starting branch cut for 1.29.0-wmf.18 |
[production] |
| 17:07 |
<godog> |
swift codfw-prod: bump ms-be2028 ms-be2039 object weight to 3000 - T158337 |
[production] |
| 17:06 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic2021.codfw.wmnet |
[production] |
| 16:39 |
<urandom> |
T111113: Restarting remaining Cassandra instances, rack 'b', codfw (restbase20{02,07,10}) |
[production] |
| 16:19 |
<urandom> |
T111113: Restarting Cassandra on restbase2001 to apply mandatory client encryption (canary) |
[production] |