|
2017-04-26
§
|
| 13:00 |
<ema> |
cp2017: restart varnish-be |
[production] |
| 12:56 |
<marostegui> |
Shutdown db1092 for maintenance - https://phabricator.wikimedia.org/T162681 |
[production] |
| 12:55 |
<gehel> |
restart elasticsearch on relforge1001 to validate new config - T161830 |
[production] |
| 12:46 |
<moritzm> |
installing mysql security updates (5.5 as packaged in Debian jessie) |
[production] |
| 12:43 |
<ema@neodymium> |
conftool action : set/pooled=no; selector: name=cp2014.codfw.wmnet,service=varnish-be |
[production] |
| 11:32 |
<jynus> |
applying new events_coredb_slave.sql on db2055 T160984 |
[production] |
| 11:31 |
<moritzm> |
rebooting mwlog2001 for update to Linux 4.9 |
[production] |
| 10:47 |
<ladsgroup@naos> |
Synchronized wmf-config/Wikibase-labs.php: T142104, part II (duration: 00m 56s) |
[production] |
| 10:45 |
<ladsgroup@naos> |
Synchronized static/images/wikibase/echoIcon.svg: T142104, part I (duration: 01m 04s) |
[production] |
| 10:44 |
<marostegui> |
Deploy alter table on s5, on db1063 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 |
[production] |
| 10:39 |
<jynus@naos> |
Synchronized wmf-config/db-eqiad.php: switch s5 eqiad master from db1049 to db1063 (duration: 01m 24s) |
[production] |
| 09:48 |
<jynus> |
migrating s5 eqiad replicas under db1063 |
[production] |
| 09:42 |
<jynus> |
restarting mariadb at db1063 |
[production] |
| 09:24 |
<marostegui> |
Shutdown db1094, db1093, db1091 for maintenance - T162681 |
[production] |
| 09:16 |
<marostegui> |
Shutdown es1019 for maintenance - T162681 |
[production] |
| 08:32 |
<elukey> |
Gracefully stopping hadoop daemons on Hadoop nodes affected by Row-D maintenance |
[production] |
| 08:29 |
<marostegui> |
Deploy alter table on change_tag and tag_summary on silver and labtestweb2001 - T147166 |
[production] |
| 08:27 |
<marostegui@naos> |
Synchronized wmf-config/db-eqiad.php: Depool hosts that need to be moved for the network maintenance - T162681 (duration: 02m 25s) |
[production] |
| 08:22 |
<moritzm> |
reimaging terbium to jessie |
[production] |
| 07:59 |
<jynus> |
shutting down mariadb on db1040 as a backup before decommissioning |
[production] |
| 07:48 |
<marostegui> |
Deploy alter table on s1, on db1052 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 |
[production] |
| 07:30 |
<marostegui> |
Deploy alter table on s7, on db1062 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 |
[production] |
| 07:24 |
<marostegui> |
Deploy alter table on s4, on db1068 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 |
[production] |
| 07:09 |
<marostegui> |
Deploy alter table on s6, on db1061 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 |
[production] |
| 06:56 |
<marostegui@naos> |
Synchronized wmf-config/db-eqiad.php: Repool db1071 - T162539 T163548 (duration: 02m 24s) |
[production] |
| 06:45 |
<marostegui> |
Deploy alter table on s2, on db1054 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 |
[production] |
| 06:10 |
<marostegui> |
Deploy alter table on s3, on db1075 (eqiad master) for tables: change_tag and tag_summary - T147166 |
[production] |
| 05:57 |
<marostegui> |
Deploy alter table enwiki.revision on labsdb1011 - T132416 |
[production] |
| 00:20 |
<catrope@naos> |
Synchronized php-1.29.0-wmf.21/extensions/Flow/modules/flow/ui/widgets/mw.flow.ui.ReplyWidget.js: T163749 (duration: 01m 24s) |
[production] |
|
2017-04-25
§
|
| 22:24 |
<mutante> |
mediawiki maintenance servers: last log entry was _before_ merging https://gerrit.wikimedia.org/r/#/c/342777/ and making a change |
[production] |
| 22:23 |
<andrewbogott> |
re-enabling dns on labservices1001 |
[production] |
| 22:22 |
<mutante> |
mediawiki maintenance servers: making wasat identical to terbium. wasat is currently the active server running crons. no change there at all. on terbium where crons are inactive, some log files were removed |
[production] |
| 22:13 |
<twentyafterfour@naos> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.29.0-wmf.21 |
[production] |
| 22:08 |
<madhuvishy> |
Reenabled labs instance creation and deletion on horizon |
[production] |
| 22:05 |
<twentyafterfour@naos> |
Finished scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #5) (duration: 21m 52s) |
[production] |
| 22:02 |
<andrewbogott> |
causing an intentional outage of labs-ns0 and labs-recursor0 to make sure we're properly girded for tomorrow's switch replacement. |
[production] |
| 21:43 |
<twentyafterfour@naos> |
Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #5) |
[production] |
| 21:41 |
<twentyafterfour@naos> |
scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_66989801"/* "/srv/mediawiki-staging/php-1.29.0-wmf.21/cache/l10n"' returned non-zero exit status 1 (duration: 03m 38s) |
[production] |
| 21:38 |
<twentyafterfour@naos> |
Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #4) |
[production] |
| 21:33 |
<twentyafterfour@naos> |
scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_930292683"/* "/srv/mediawiki-staging/php-1.29.0-wmf.21/cache/l10n"' returned non-zero exit status 1 (duration: 03m 46s) |
[production] |
| 21:30 |
<twentyafterfour@naos> |
Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #3) |
[production] |
| 21:23 |
<twentyafterfour@naos> |
scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_2414756836"/* "/srv/mediawiki-staging/php-1.29.0-wmf.21/cache/l10n"' returned non-zero exit status 1 (duration: 00m 54s) |
[production] |
| 21:23 |
<twentyafterfour@naos> |
Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 (attempt #2) |
[production] |
| 21:09 |
<twentyafterfour@naos> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3498979833" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 01m 56s) |
[production] |
| 21:07 |
<twentyafterfour@naos> |
Started scap: sync 1.29.0-wmf.21 to testwikis (pre-group0) refs T161733 |
[production] |
| 20:00 |
<madhuvishy> |
Labs instance creation and deletion on horizon temporarily disabled via https://gerrit.wikimedia.org/r/350266 |
[production] |
| 19:50 |
<demon@naos> |
Synchronized wmf-config/CommonSettings-labs.php: no-op, beta change (duration: 01m 58s) |
[production] |
| 18:55 |
<chasemp> |
restart nova-fullstack on labnet1001 |
[production] |
| 18:50 |
<chasemp> |
downtime labservices1001 as we fail away from it and puppet staleness on labservices1002 |
[production] |
| 18:38 |
<andrewbogott> |
disabling nova-api for another try at labservices failover |
[production] |