|
2020-10-21
§
|
| 15:00 |
<otto@deploy1001> |
Started deploy [analytics/refinery@e4d16f0] (hadoop-test): deploying with updated camus to test cluster |
[production] |
| 14:56 |
<crusnov@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:44 |
<reedy@deploy1001> |
Synchronized wmf-config/wikitech.php: Set CURLOPT_RETURNTRANSFER true in gerrit handler T242554 (duration: 01m 07s) |
[production] |
| 14:34 |
<dcausse> |
restarting blazegraph on codfw servers (T263952) |
[production] |
| 13:21 |
<moritzm> |
pooling ldap-replica2003 T264388 |
[production] |
| 13:04 |
<liw@deploy1001> |
Synchronized php: group1 wikis to 1.36.0-wmf.14 (duration: 01m 04s) |
[production] |
| 13:03 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.14 |
[production] |
| 11:40 |
<matthiasmullie> |
EU B&C done |
[production] |
| 11:33 |
<mlitn@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [WikibaseMediaInfo] Add config for related terms API (duration: 01m 04s) |
[production] |
| 11:17 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: 785404fa2b998947d236aebe481ee1abcbd14220: Disable registrations stat on Special:TranslationStats (T264158) (duration: 01m 05s) |
[production] |
| 11:10 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 11567427c3f7d2908b29046ee56a7b0c0da32c09: Enable ContentTranslation in 5 Wikipedias as a default tool (T264737; T264738; T264739; T264740; T264741) (duration: 01m 30s) |
[production] |
| 11:00 |
<marostegui> |
Upgrade db2093's mariadb version T266003 |
[production] |
| 10:58 |
<aborrero@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:56 |
<aborrero@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:38 |
<Urbanecm> |
Start of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` (wiki=rowiki; T246539) |
[production] |
| 10:37 |
<Urbanecm> |
End of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` (wiki=srwiki; T246539) |
[production] |
| 10:01 |
<Urbanecm> |
Start of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` (wiki=srwiki; T246539) |
[production] |
| 10:00 |
<Urbanecm> |
End of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` (wiki=nowiki; T246539) |
[production] |
| 09:59 |
<vgutierrez> |
Bump ECDHE-ECDSA-AES128-SHA pageview replacement to 100% - T258405 |
[production] |
| 09:42 |
<Urbanecm> |
Start of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` (wiki=nowiki; T246539) |
[production] |
| 09:42 |
<Urbanecm> |
End of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` (wiki=shwiki; T246539) |
[production] |
| 09:38 |
<Urbanecm> |
Start of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` (wiki=shwiki; T246539) |
[production] |
| 09:37 |
<Urbanecm> |
mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log # wiki=warwiki; T246539 |
[production] |
| 09:30 |
<Urbanecm> |
End of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` (wiki=viwiki; T246539) |
[production] |
| 09:23 |
<aborrero@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:22 |
<root@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 09:21 |
<aborrero@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:51 |
<Urbanecm> |
Start of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` (wiki=viwiki; T246539) |
[production] |
| 08:50 |
<Urbanecm> |
mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log # wiki=cebwiki; T246539 |
[production] |
| 08:46 |
<Urbanecm> |
[urbanecm@mwmaint2001 ~/updateVarDumps/output/group2-medium/output]$ mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=apiportalwiki # T246539 |
[production] |
| 08:38 |
<root@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 08:38 |
<root@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 08:38 |
<root@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 08:33 |
<godog> |
swift codfw-prod: bump object weight for ms-be2057 - T261633 |
[production] |
| 08:10 |
<XioNoX> |
Upgrade Routinator 3000 to 0.8.0 on rpki1001 - T266001 |
[production] |
| 08:09 |
<XioNoX> |
add Routinator 3000 0.8.0 to apt - T266001 |
[production] |
| 07:58 |
<elukey> |
update analytics-in4 filter on cr1/cr2-eqiad for https://gerrit.wikimedia.org/r/635319 |
[production] |
| 04:35 |
<ryankemper> |
re-enabled icinga notifications on all wdqs hosts now that `wdqs-updater` is healthy |
[production] |
|
2020-10-20
§
|
| 22:10 |
<dwisehaupt> |
frmon2001 upgraded to buster with grafana 7.2.1 |
[production] |
| 21:19 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 21:18 |
<cdanis> |
✔️ cdanis@mw2252.codfw.wmnet ~ 🕠🍺 sudo depool |
[production] |
| 20:57 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@e4d16f0] (thin): Regular analytics weekly train THIN [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] (duration: 00m 08s) |
[production] |
| 20:56 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@e4d16f0] (thin): Regular analytics weekly train THIN [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] |
[production] |
| 20:39 |
<cdanis> |
doing some manual testing on mw2221, depooled and puppet disabled |
[production] |
| 20:33 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@e4d16f0]: Regular analytics weekly train [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] (duration: 08m 10s) |
[production] |
| 20:31 |
<ryankemper> |
[Temporarily] disabled notifications for all wdqs hosts while we figure out how to unstick the updater process. Impact is that new updates will be delayed, but queries will still keep serving as normal, so fixing this is a priority but note that there's no availability outage |
[production] |
| 20:29 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 20:25 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@e4d16f0]: Regular analytics weekly train [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] |
[production] |
| 20:19 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 20:18 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |