|
2019-02-07
§
|
| 12:34 |
<marostegui> |
Powercycle mw1299 as it is down and not responding |
[production] |
| 12:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1101 after alter and mysql upgrade (duration: 03m 02s) |
[production] |
| 12:26 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/interwiki.php: SWAT: [[gerrit:488903|Update interwiki cache to have yuewiktionary instead of zh-yue (T214400)]] (duration: 03m 04s) |
[production] |
| 12:06 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet |
[production] |
| 12:03 |
<arturo> |
T214448 reimaging again cloudvirt200[1-3]-dev.codfw.wmnet |
[production] |
| 11:55 |
<marostegui> |
Stop MySQL on db1101:3317 and db1101:3318 for mysql upgrade |
[production] |
| 11:37 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2055 (duration: 03m 02s) |
[production] |
| 11:17 |
<fsero> |
upgrade helm to 2.12.2 on deploy{1001,2001} and contint{1001,2001} T215244 |
[production] |
| 11:16 |
<fsero> |
upgrade helm to 2.12.2 on deploy{1001,2001} and contint{1001,2001} |
[production] |
| 10:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1101 for alter and mysql upgrade (duration: 00m 56s) |
[production] |
| 10:43 |
<marostegui> |
Run mysqldump from dbstore1003 to dump dbstore1002:staging.mep_word_persistence - T215450 |
[production] |
| 09:49 |
<marostegui> |
Deploy schema change on db1116 - T210713 |
[production] |
| 09:41 |
<akosiaris> |
reboot mwdebug1001, mwdebug1002, mwdebug2001, mwdebug2002 for VCPU upgrade. T212955 |
[production] |
| 09:23 |
<jynus> |
running alter table on db2055 for perforamance testing T212092 |
[production] |
| 09:15 |
<fsero> |
uploading helm and tiller 2.12.2 deb package to stretch and jessie |
[production] |
| 08:53 |
<marostegui> |
Deploy schema change on s7 codfw master (db2047), this will generate lag on s7 codfw - T210713 |
[production] |
| 08:34 |
<godog> |
swift codfw-prod: more weight to ms-be2047 - T209395 T209921 |
[production] |
| 08:14 |
<marostegui> |
Deploy schema change on s4 primary master (db1068) - T210713 |
[production] |
| 08:13 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1081 (duration: 00m 54s) |
[production] |
| 07:50 |
<marostegui> |
Deploy schema change on db1081 |
[production] |
| 07:49 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1081 (duration: 00m 53s) |
[production] |
| 07:48 |
<reedy@deploy1001> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 20s) |
[production] |
| 07:43 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 53s) |
[production] |
| 07:42 |
<reedy@deploy1001> |
Synchronized dblists/: Wikimania T215486 (duration: 00m 54s) |
[production] |
| 07:03 |
<marostegui> |
Deploy schema change on db1084 - T210713 |
[production] |
| 07:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 55s) |
[production] |
| 06:48 |
<marostegui> |
Restore consistency options on db2051 |
[production] |
| 06:14 |
<marostegui> |
Ease consistency options on db2051 (s4 master) to let it catch up on replication |
[production] |
| 04:35 |
<tstarling@deploy1001> |
Synchronized wmf-config/set-time-limit.php: (no justification provided) (duration: 00m 54s) |
[production] |
| 04:00 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Disable EP namespaces on wikis with no EP pages (duration: 00m 57s) |
[production] |
| 01:31 |
<eileen> |
civicrm revision changed from c5aec3ae76 to 08be00e87f, config revision is 306b4de48f |
[production] |
| 01:24 |
<eileen> |
civicrm revision changed from 6161a021c0 to c5aec3ae76, config revision is 306b4de48f |
[production] |
| 01:05 |
<twentyafterfour> |
US Evening SWAT is complete |
[production] |
| 01:04 |
<twentyafterfour> |
no phabricator deployment tonight |
[production] |
| 01:04 |
<eileen> |
civicrm revision changed from 613b388916 to 6161a021c0, config revision is 306b4de48f |
[production] |
| 00:56 |
<twentyafterfour@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT config change for Bug: T214003 (duration: 00m 53s) |
[production] |
| 00:53 |
<twentyafterfour@deploy1001> |
Synchronized php-1.33.0-wmf.16/extensions/VisualEditor/: SWAT f89e12fc466d2c51343d9815c70a0b4602acc333 to fix bug: T209610 (duration: 00m 55s) |
[production] |
| 00:48 |
<twentyafterfour@deploy1001> |
Synchronized php-1.33.0-wmf.16/extensions/MobileFrontend/: SWAT dd8654ac9b3f2e88241e65d3ea35aea9699defc5 (duration: 01m 00s) |
[production] |
| 00:47 |
<twentyafterfour> |
syncing commit dd8654ac9b3f2e88241e65d3ea35aea9699defc5 for Bug: T209052 |
[production] |
| 00:24 |
<twentyafterfour> |
running `mwscript migrateUserGroup.php commonswiki extended-uploader autopatrolled` on deploy1001 |
[production] |
|
2019-02-06
§
|
| 23:58 |
<mutante> |
restarting icinga on icinga1001 to pick up new check command ? |
[production] |
| 22:22 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.33.0-wmf.16 refs T206670 (duration: 00m 53s) |
[production] |
| 22:21 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.16 refs T206670 |
[production] |
| 21:45 |
<mutante> |
LDAP - adding brennen to wmf, releng, ciadmin - Welcome Brennan Bearnes, Software Engineer in Release Engineering (T215365 T214556) |
[production] |
| 21:05 |
<arlolra@deploy1001> |
Finished deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71 (duration: 03m 43s) |
[production] |
| 21:04 |
<Krinkle> |
krinkle@webperf1001 Kill xenon-log (pid 449). It seems its Redis TCP socket to mwlog1001 has been stuck since Dec 13, causing the process to indefinitely hang on listen()/socket.recv() |
[production] |
| 21:01 |
<arlolra@deploy1001> |
Started deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71 |
[production] |
| 20:49 |
<mutante> |
LDAP - adding h78na to wmf - welcome Hana Worku, developer on the multimedia team (T215352) |
[production] |
| 20:40 |
<mutante> |
LDAP - adding egardner to wmf - welcome Eric Gardner , software engineer in Audiences (T214654) |
[production] |
| 20:35 |
<twentyafterfour> |
1.33.0-wmf.16 has a significantly higher rate of "entire web request took longer than 60 seconds and timed out" |
[production] |