|
2017-11-30
§
|
| 13:18 |
<herron> |
cutting codfw puppet agents over to puppetmaster2001.codfw.wmnet |
[production] |
| 12:07 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Tackle s4 DB weights to make them more equal (duration: 00m 48s) |
[production] |
| 11:58 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1081 and reduce traffic for db1091 (duration: 00m 50s) |
[production] |
| 10:33 |
<jynus> |
stopping replication on db1044 and db1095 (s3) |
[production] |
| 09:16 |
<moritzm> |
installing exim security updates on stretch (jessie/trusty not affected) |
[production] |
| 09:14 |
<elukey> |
drain and reboot analytics1029/1030 for jvm+kernel updates (Hadoop worker canaries) |
[production] |
| 09:05 |
<godog> |
add 200G of space to graphite2002 carbon lv |
[production] |
| 08:25 |
<moritzm> |
rolling restart of mw canaries to pick up curl security update |
[production] |
| 08:21 |
<marostegui> |
Enable GTID on s8 eqiad hosts that do not have it enabled (db1109, db1104, db1101, db1092, db1087, db1063) - T177208 |
[production] |
| 07:48 |
<moritzm> |
installing curl security updates |
[production] |
| 07:01 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1056 (duration: 01m 18s) |
[production] |
| 06:48 |
<marostegui> |
Deploy schema change on dbstore1001 s3 - T174569 |
[production] |
| 06:26 |
<marostegui> |
Deploy schema change on s3 db1077 - T174569 |
[production] |
| 02:48 |
<ebernhardson@tin> |
Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 03m 15s) |
[production] |
| 02:45 |
<ebernhardson@tin> |
Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) |
[production] |
| 02:29 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.8) (duration: 08m 43s) |
[production] |
| 01:17 |
<ejegg> |
updated fundraising tools from 6d4b6f388e64ff4dc9a6a501bc8339fc1294fe9f to 6e604fd9f3ca57935e2c35e37b7e85b4ca438d6a |
[production] |
| 01:06 |
<twentyafterfour> |
finished phabricator upgrade, service is online and appears to be functioning normally |
[production] |
| 01:03 |
<twentyafterfour> |
Starting phabricator upgrade & maintenance. Service will be offline for less than 5 minutes. |
[production] |
| 01:02 |
<bsitzmann@tin> |
Finished deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 (T181004) (duration: 06m 08s) |
[production] |
| 00:56 |
<bsitzmann@tin> |
Started deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 (T181004) |
[production] |
| 00:53 |
<maxsem@tin> |
Finished scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/ (duration: 26m 38s) |
[production] |
| 00:48 |
<ejegg> |
re-enabled CiviCRM jobs |
[production] |
| 00:42 |
<ejegg> |
updated CiviCRM from 0f95f3e37f9651a5fd262bf1f5e39929abee0d88 to e81228f79ee4a70b7a72e8aafbb5ee387da4230d |
[production] |
| 00:40 |
<ejegg> |
disabled CiviCRM jobs |
[production] |
| 00:31 |
<ejegg> |
updated fundraising dashboard from 6ee656759561d524c1ed8a15ac4da4d0fce887a7 to 114131713e566069575f481281c74349e6bba336 |
[production] |
| 00:27 |
<maxsem@tin> |
Started scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/ |
[production] |
| 00:24 |
<maxsem@tin> |
Synchronized php-1.31.0-wmf.10/extensions/ContentTranslation/: https://gerrit.wikimedia.org/r/#/c/394206/ (duration: 00m 52s) |
[production] |
| 00:21 |
<ejegg> |
added weekly Ingenico audit processing job in makemissing mode |
[production] |
|
2017-11-29
§
|
| 23:20 |
<eileen> |
update civicrm from a1022cf2a64c048382a1ae5da7f1f81da0336d62 to 0f95f3e37f9651a5fd262bf1f5e39929abee0d88 (Benevity import fix) |
[production] |
| 22:23 |
<hashar> |
Nodepool had some troubles spawning new instances from 21:09 to 21:36, and took a while to recover. Issue similar to T170492#3581822 |
[production] |
| 21:51 |
<SMalyshev> |
starting wikidata reindex (T181426) |
[production] |
| 21:23 |
<no_justification> |
docroot sync was for If1afa59a |
[production] |
| 21:21 |
<demon@tin> |
Synchronized docroot/: (no justification provided) (duration: 00m 49s) |
[production] |
| 20:29 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.10 |
[production] |
| 20:26 |
<demon@tin> |
Synchronized php: symlink bump (duration: 00m 48s) |
[production] |
| 20:05 |
<urandom> |
restarting cassandra bootstrap of restbase1012-a (T179422) |
[production] |
| 19:55 |
<mutante> |
restarting gerrit to apply config change and set gitBasicAuth to true to unblock T171758 (gerrit:391865) |
[production] |
| 19:50 |
<herron> |
beginning rolling cut over of eqiad scb hosts to codfw puppet 4 masters |
[production] |
| 19:15 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/394059/3 (duration: 00m 49s) |
[production] |
| 18:57 |
<herron> |
beginning cutover of codfw lvs2* systems to codfw puppet 4 masters. first standby nodes, then active nodes |
[production] |
| 18:12 |
<marostegui> |
Compress s5 on db1096 - T178359 |
[production] |
| 18:10 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 53s) |
[production] |
| 18:05 |
<awight@tin> |
Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 01m 11s) |
[production] |
| 18:04 |
<awight@tin> |
Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster |
[production] |
| 18:03 |
<herron> |
beginning cut over of codfw cp2* servers to codfw puppet 4 masters |
[production] |
| 17:40 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1110 with low load (duration: 00m 48s) |
[production] |
| 17:40 |
<godog> |
bootstrapping restbase1012-a - T179422 |
[production] |
| 17:19 |
<herron> |
beginning cut over of cp200[12] to codfw puppet 4 masters |
[production] |
| 17:16 |
<ejegg> |
increased time limit for donation queue consumer from 60 to 70 seconds, reduced thank you job from 55 to 45 seconds |
[production] |