|
2018-05-25
§
|
| 11:21 |
<akosiaris> |
rebalance row_B codfw ganeti nodegroup. Cluster is now fully upgraded to stretch |
[production] |
| 11:18 |
<akosiaris> |
powercycling ms-be1034, box is unresposive, tons of logs "sd 0:1:0:1: rejecting I/O to offline device" |
[production] |
| 10:37 |
<XioNoX> |
test force mtu 1400 between cp1074 and cp3039 - T195365 |
[production] |
| 09:20 |
<marostegui> |
Deploy schema change on db1105:3311 - T190148 T191519 T188299 |
[production] |
| 09:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 01m 20s) |
[production] |
| 09:16 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 01m 19s) |
[production] |
| 09:10 |
<marostegui@tin> |
scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 09:03 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 01m 20s) |
[production] |
| 08:44 |
<marostegui> |
Stop MySQL on db1120 to transfer its content to db1125 - T190704 |
[production] |
| 08:36 |
<marostegui> |
Add tmp1 back on db1092 - https://phabricator.wikimedia.org/T194273 |
[production] |
| 08:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 20s) |
[production] |
| 08:30 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 01m 20s) |
[production] |
| 08:26 |
<gilles@tin> |
Synchronized wmf-config/InitialiseSettings.php: T187299 Launch performance survey on frwiki (duration: 01m 22s) |
[production] |
| 07:05 |
<jynus> |
stop db1117:m2 to clone it to db1065 |
[production] |
| 06:57 |
<marostegui> |
Deploy schema change on db1114 - T190148 T191519 T188299 |
[production] |
| 06:57 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 01m 20s) |
[production] |
| 06:53 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 01m 20s) |
[production] |
| 06:37 |
<jynus> |
reimage db1065 after raid rebuild |
[production] |
| 05:57 |
<marostegui> |
Add tmp1 index back on dbstore1002 - T194273 |
[production] |
| 05:25 |
<marostegui> |
Stop MySQL on db1116 to copy its content to db1124 - T190704 |
[production] |
| 05:23 |
<marostegui> |
Deploy schema change on db1089 - T190148 T191519 T188299 |
[production] |
| 05:23 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 01m 20s) |
[production] |
| 05:17 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 01m 21s) |
[production] |
| 05:05 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 01m 20s) |
[production] |
| 05:05 |
<marostegui> |
Add tmp1 index back to db1109 - T194273 |
[production] |
| 04:59 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 01m 21s) |
[production] |
| 03:10 |
<papaul> |
OS install on db209[4-5] |
[production] |
|
2018-05-24
§
|
| 23:13 |
<ladsgroup@tin> |
Synchronized php-1.32.0-wmf.5/extensions/Wikibase: Dumps: Allow several --entity-type arguments (T195420) (duration: 02m 25s) |
[production] |
| 22:21 |
<ladsgroup@tin> |
Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520) (duration: 01m 20s) |
[production] |
| 22:11 |
<ladsgroup@tin> |
Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520) (duration: 01m 21s) |
[production] |
| 20:58 |
<marostegui> |
Add tmp1 index back on db1104 |
[production] |
| 20:57 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 01m 20s) |
[production] |
| 20:56 |
<legoktm@tin> |
Synchronized php-1.32.0-wmf.5/extensions/VisualEditor/: no-op, sync with git state (duration: 01m 21s) |
[production] |
| 20:54 |
<legoktm@tin> |
Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: no-op, sync with git state (duration: 01m 20s) |
[production] |
| 20:41 |
<legoktm@tin> |
Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: no-op, sync with git state (duration: 01m 20s) |
[production] |
| 20:17 |
<legoktm@tin> |
Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: wmf.5 this time (duration: 01m 19s) |
[production] |
| 20:14 |
<legoktm@tin> |
Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Add debug logging (duration: 01m 19s) |
[production] |
| 20:11 |
<legoktm@tin> |
Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Disable TermSqlIndex::getMatchingTerms (duration: 01m 20s) |
[production] |
| 19:49 |
<ejegg> |
updated payments-wiki from c81e25f8d3 to 43989ebc96 |
[production] |
| 19:44 |
<reedy@tin> |
Synchronized wmf-config/Wikibase.php: Disable PropSuggester (duration: 01m 21s) |
[production] |
| 19:30 |
<twentyafterfour@tin> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 19:14 |
<twentyafterfour> |
Starting the MediaWiki train for Thursday May 24, today I will be deploying wmf/1.32.0-wmf.5 to all wikis |
[production] |
| 18:31 |
<mutante> |
netmon1002, netmon2001: systemctl mask uwsgi; systemctl reset-failed - to fix Icinga alert about broken DPKG since last netbox deploy and to match existing status on labtestweb2001 |
[production] |
| 18:21 |
<XioNoX> |
repool esams |
[production] |
| 18:01 |
<no_justification> |
gerrit restarting on cobalt, back soon |
[production] |
| 17:50 |
<mutante> |
mwdebug1001 - apt-get remove to clean up packages in "non ii" states |
[production] |
| 16:59 |
<anomie> |
Running populateExternallinksIndex60.php on group 1 for T59176 |
[production] |
| 16:50 |
<XioNoX> |
depooled esams - investigating issues |
[production] |
| 16:31 |
<anomie> |
Running deduplicateArchiveRevId.php on group 2 for T193180 |
[production] |
| 16:28 |
<jynus> |
manually failover the backup host for m2 to db1117:3322 |
[production] |