51-100 of 10000 results (52ms)
2018-01-10 §
13:26 <_joe_> restarting pybal on lvs2003 [production]
13:03 <mobrovac@tin> Finished deploy [restbase/deploy@a2aabfb]: API: add top-by-country, change recommendation route, fix duplicates in onthisday - T181520 T170877 T175974 (duration: 08m 00s) [production]
12:55 <mobrovac@tin> Started deploy [restbase/deploy@a2aabfb]: API: add top-by-country, change recommendation route, fix duplicates in onthisday - T181520 T170877 T175974 [production]
12:54 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 - T174569 (duration: 01m 03s) [production]
12:54 <marostegui> Deploy schema change on db1097:3315 - https://phabricator.wikimedia.org/T174569 [production]
12:46 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1106 - T174569 (duration: 01m 03s) [production]
12:38 <moritzm> migrating instances off ganeti2004 for subsequent reboot for kernel security update [production]
12:19 <moritzm> migrating instances off ganeti2005 for subsequent reboot for kernel security update [production]
12:11 <moritzm> rebooting einsteinium for kernel security update [production]
11:51 <moritzm> migrating instances off ganeti2006 for subsequent reboot for kernel security update [production]
11:45 <godog> downtime decomissioned restbase cassandra 2 hosts [production]
11:39 <moritzm> rebooting mw1201-mw1208 for kernel security update (along with update to HHVM 3.18.6) [production]
11:33 <marostegui> Deploy schema change on db1106 - T174569 [production]
11:26 <elukey> reboot analytics1044->47 for kernel updates [production]
11:23 <moritzm> migrating instances off ganeti2007 for subsequent reboot for kernel security update [production]
11:19 <volans> Icinga failover to tegmen completed - T170353 [production]
11:12 <moritzm> migrating instances off ganeti2008 for subsequent reboot for kernel security update [production]
11:07 <volans> start failovering of Icinga to tegmen - T170353 [production]
10:55 <elukey> reboot analytics1040->43 for kernel updates [production]
10:29 <godog> reimage restbase1011 to test HBA mode - T184100 [production]
10:16 <moritzm> rebooting bast4001 for kernel security update [production]
10:06 <elukey> rebooting analytics1035 (hadoop worker node and hdfs journal node) for kernel updates [production]
10:02 <moritzm> rebooting tegmen for kernel security update [production]
09:50 <godog> shut cassandra 2 on restbase legacy nodes - T184100 [production]
09:40 <moritzm> rebooting kubernetes workers (plus staging hosts) for kernel security update [production]
09:39 <ema> eqiad LVSs: upgrade to latest jessie point release (8.10) T182656 and linux kernel 4.9.65-3+deb9u1~bpo8+2 (KPTI) T184267 [production]
09:32 <marostegui> Upgrade kernel on db1067 [production]
09:27 <godog> stop restbase on cassandra 2 nodes - T184100 [production]
09:15 <marostegui> Deploy schema change on db1051 - T174569 [production]
09:12 <moritzm> rebooting radium (tor relay) for kernel security update [production]
08:42 <marostegui> Stop replication in sync on db1089 and db1067 - T162807 [production]
08:41 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1067 and db1089 - T162807 (duration: 01m 05s) [production]
08:38 <marostegui> Deploy schema change on s5 dbstore1001 - T174569 [production]
08:33 <moritzm> rebooting mw1299-mw1306 (job runners) for kernel security update (along with update to HHVM 3.18.6) [production]
08:28 <hashar> contint1001: upgraded Zuul 2.5.0-8-gcbc7f62-wmf4jessie1 .. 2.5.0-8-gcbc7f62-wmf6 | T158243 [production]
08:13 <marostegui> Deploy schema change on s5 dbstore1002 - T174569 [production]
07:44 <moritzm> rebooting mw1262-mw1275 for kernel security update (along with update to HHVM 3.18.6) [production]
07:37 <marostegui> Drop external_user from wikidatawiki - T184247 [production]
06:17 <marostegui> Deploy schema change on s5 codfw master (db2052) with replication (this will generate lag on codfw) - T174569 [production]
02:24 <l10nupdate@tin> scap sync-l10n completed (1.31.0-wmf.15) (duration: 06m 02s) [production]
01:39 <mutante> mw1226 - high load - hhvm-dump-debug > /root/hhvm-dump-debug-20170109-1739PST.log ; restart-hhvm [production]
00:43 <mutante> rebooting gerrit server for kernel upgrade [production]
00:18 <mutante> rebooting phabricator server for kernel upgrade [production]
2018-01-09 §
22:52 <godog> ms-be1033 truncate unrotated and big server.log [production]
22:22 <aaron@tin> Synchronized php-1.31.0-wmf.16/includes/Setup.php: 68b4bbfbc12c626 (duration: 01m 15s) [production]
22:20 <mutante> netmon2001 - arming keyholder for rancid [production]
21:10 <mepps> updated SmashPig from 45aa62650c to 778e8f87b4 [production]
20:57 <twentyafterfour@tin> Finished scap: Deploy 1.31.0-wmf.16 to test wikis and rebuild l10n. refs T180749 (attempt 2) (duration: 36m 34s) [production]
20:21 <twentyafterfour@tin> Started scap: Deploy 1.31.0-wmf.16 to test wikis and rebuild l10n. refs T180749 (attempt 2) [production]
20:14 <twentyafterfour@tin> scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="test2wiki" --outdir="/tmp/scap_l10n_3984299293" --threads=10 --lang en --quiet' returned non-zero exit status 1 (duration: 02m 44s) [production]