651-700 of 10000 results (33ms)
2017-04-26 §
10:47 <ladsgroup@naos> Synchronized wmf-config/Wikibase-labs.php: T142104, part II (duration: 00m 56s) [production]
10:45 <ladsgroup@naos> Synchronized static/images/wikibase/echoIcon.svg: T142104, part I (duration: 01m 04s) [production]
10:44 <marostegui> Deploy alter table on s5, on db1063 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 [production]
10:39 <jynus@naos> Synchronized wmf-config/db-eqiad.php: switch s5 eqiad master from db1049 to db1063 (duration: 01m 24s) [production]
09:48 <jynus> migrating s5 eqiad replicas under db1063 [production]
09:42 <jynus> restarting mariadb at db1063 [production]
09:24 <marostegui> Shutdown db1094, db1093, db1091 for maintenance - T162681 [production]
09:16 <marostegui> Shutdown es1019 for maintenance - T162681 [production]
08:32 <elukey> Gracefully stopping hadoop daemons on Hadoop nodes affected by Row-D maintenance [production]
08:29 <marostegui> Deploy alter table on change_tag and tag_summary on silver and labtestweb2001 - T147166 [production]
08:27 <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Depool hosts that need to be moved for the network maintenance - T162681 (duration: 02m 25s) [production]
08:22 <moritzm> reimaging terbium to jessie [production]
07:59 <jynus> shutting down mariadb on db1040 as a backup before decommissioning [production]
07:48 <marostegui> Deploy alter table on s1, on db1052 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 [production]
07:30 <marostegui> Deploy alter table on s7, on db1062 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 [production]
07:24 <marostegui> Deploy alter table on s4, on db1068 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 [production]
07:09 <marostegui> Deploy alter table on s6, on db1061 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 [production]
06:56 <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Repool db1071 - T162539 T163548 (duration: 02m 24s) [production]
06:45 <marostegui> Deploy alter table on s2, on db1054 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 [production]
06:10 <marostegui> Deploy alter table on s3, on db1075 (eqiad master) for tables: change_tag and tag_summary - T147166 [production]
05:57 <marostegui> Deploy alter table enwiki.revision on labsdb1011 - T132416 [production]
00:20 <catrope@naos> Synchronized php-1.29.0-wmf.21/extensions/Flow/modules/flow/ui/widgets/mw.flow.ui.ReplyWidget.js: T163749 (duration: 01m 24s) [production]
2017-04-25 §
22:24 <mutante> mediawiki maintenance servers: last log entry was _before_ merging https://gerrit.wikimedia.org/r/#/c/342777/ and making a change [production]
22:23 <andrewbogott> re-enabling dns on labservices1001 [production]
22:22 <mutante> mediawiki maintenance servers: making wasat identical to terbium. wasat is currently the active server running crons. no change there at all. on terbium where crons are inactive, some log files were removed [production]
22:13 <twentyafterfour@naos> rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.29.0-wmf.21 [production]
22:08 <madhuvishy> Reenabled labs instance creation and deletion on horizon [production]
22:05 <twentyafterfour@naos> Finished scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #5) (duration: 21m 52s) [production]
22:02 <andrewbogott> causing an intentional outage of labs-ns0 and labs-recursor0 to make sure we're properly girded for tomorrow's switch replacement. [production]
21:43 <twentyafterfour@naos> Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #5) [production]
21:41 <twentyafterfour@naos> scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_66989801"/* "/srv/mediawiki-staging/php-1.29.0-wmf.21/cache/l10n"' returned non-zero exit status 1 (duration: 03m 38s) [production]
21:38 <twentyafterfour@naos> Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #4) [production]
21:33 <twentyafterfour@naos> scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_930292683"/* "/srv/mediawiki-staging/php-1.29.0-wmf.21/cache/l10n"' returned non-zero exit status 1 (duration: 03m 46s) [production]
21:30 <twentyafterfour@naos> Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #3) [production]
21:23 <twentyafterfour@naos> scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_2414756836"/* "/srv/mediawiki-staging/php-1.29.0-wmf.21/cache/l10n"' returned non-zero exit status 1 (duration: 00m 54s) [production]
21:23 <twentyafterfour@naos> Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #2) [production]
21:09 <twentyafterfour@naos> scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3498979833" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 01m 56s) [production]
21:07 <twentyafterfour@naos> Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 [production]
20:00 <madhuvishy> Labs instance creation and deletion on horizon temporarily disabled via https://gerrit.wikimedia.org/r/350266 [production]
19:50 <demon@naos> Synchronized wmf-config/CommonSettings-labs.php: no-op, beta change (duration: 01m 58s) [production]
18:55 <chasemp> restart nova-fullstack on labnet1001 [production]
18:50 <chasemp> downtime labservices1001 as we fail away from it and puppet staleness on labservices1002 [production]
18:38 <andrewbogott> disabling nova-api for another try at labservices failover [production]
18:33 <twentyafterfour> Deployment Train: Branching mediawiki wmf/1.29.0-wmf.21 from master refs T161733 [production]
17:36 <jynus> running test schema change on etwiki on eqiad (depooled) T17441 [production]
17:35 <RainbowSprinkles> gerrit: Quick reboot to pick up new bouncycastle library [production]
17:25 <arlolra> Updated Parsoid to 55b90511 (T153885, T163330, T89262, T154709, T162919, T161306) [production]
17:20 <moritzm> rebooting ruthenium for update to Linux 4.9 [production]
17:19 <otto@naos> Finished deploy [eventlogging/eventbus@e7da0cc]: (no justification provided) (duration: 00m 07s) [production]
17:19 <otto@naos> Started deploy [eventlogging/eventbus@e7da0cc]: (no justification provided) [production]