|
2020-05-29
§
|
| 11:01 |
<ema> |
upload prometheus-rdkafka-exporter 0.2 to buster-wikimedia T253551 |
[production] |
| 10:53 |
<moritzm> |
updating mwdebug2002 to 7.2.31 |
[production] |
| 10:02 |
<marostegui> |
Compress InnoDB on db1138 T232446 |
[production] |
| 08:30 |
<godog> |
update swift uid/gid on thanos hosts - T123918 |
[production] |
| 08:04 |
<mutante> |
phabricator - restarted apache2 - back for me now |
[production] |
| 08:03 |
<XioNoX> |
add new AMS-IX link to LACP bundle |
[production] |
| 08:01 |
<mutante> |
phabricator - broken due to "PhabricatorRepositoryMirrorEngine::pushToGitRepository" starting git process that uses 100% CPU, stopped phd service |
[production] |
| 07:56 |
<mutante> |
phabricator - killed pid 25070 (git) which used 100% of CPU, restarted phd service |
[production] |
| 07:25 |
<moritzm> |
updating perf on buster systems to new version from 10.4 point release |
[production] |
| 07:15 |
<moritzm> |
installing el-api update from latest Buster point release |
[production] |
| 07:12 |
<moritzm> |
installing xdg-utils update from latest Buster point release |
[production] |
| 07:11 |
<mutante> |
mw1293 (canary jobrunner ) replace apache2.conf with version from mwdebug1001, restart apache, to debug for T190111 |
[production] |
| 07:00 |
<moritzm> |
installing rake security updates |
[production] |
| 06:36 |
<mutante> |
deneb - systemctl start docker-reporter-releng-images |
[production] |
| 05:20 |
<marostegui> |
Deploy schema change on db1138 (no longer s4 master) - T250055 |
[production] |
| 05:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1081 to s4 master and remove read-only from s4 T253808', diff saved to https://phabricator.wikimedia.org/P11334 and previous config saved to /var/cache/conftool/dbconfig/20200529-050224-marostegui.json |
[production] |
| 05:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s4 as read-only for maintenance T253808', diff saved to https://phabricator.wikimedia.org/P11333 and previous config saved to /var/cache/conftool/dbconfig/20200529-050153-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s4 failover from db1138 to db1081 -T253808 |
[production] |
| 04:25 |
<marostegui> |
Start topology changes in s4 - T253808 |
[production] |
|
2020-05-28
§
|
| 23:48 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.34/skins/Vector/resources/skins.vector.styles/Menu.less: T253912 Hotfix: Cannot rename emptyPortlet to empty-portlet yet (duration: 00m 59s) |
[production] |
| 22:41 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.34/extensions/WikibaseMediaInfo/src/Services/FilePageLookup.php: T253792 Follow-up 1827c7a: Ensure inNamespace() is called only on Title object (duration: 00m 58s) |
[production] |
| 22:24 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T253821 Update MachineVision block list for 2020-05-27 (duration: 00m 57s) |
[production] |
| 22:09 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Move one CheckUser right change next to the other (duration: 00m 57s) |
[production] |
| 22:06 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Remove version wrapper around wgOverrideUcfirstCharacters; always true (duration: 00m 59s) |
[production] |
| 21:48 |
<jforrester@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.34 |
[production] |
| 21:26 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.34/includes/filerepo/FileRepo.php: T253922 Mark two FileRepo functions public (duration: 01m 07s) |
[production] |
| 21:12 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.34/includes/specials/SpecialUserrights.php: T253909 Restore visibility (previously implicitely public) (duration: 01m 06s) |
[production] |
| 20:38 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.32/skins/Vector/resources/skins.vector.styles: T253905 HOTFIX: Do not apply p-personal absolute positioning to all menus (duration: 01m 07s) |
[production] |
| 20:22 |
<shdubsh> |
restart varnishmtail and atsmtail eqsin |
[production] |
| 20:11 |
<shdubsh> |
restart ncredirmtail on ncredir5001 |
[production] |
| 19:20 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: roll back the train due to T253905 |
[production] |
| 19:20 |
<twentyafterfour> |
group2 back to wmf.32 due to T253905 |
[production] |
| 19:20 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@f6d73c8] (thin): Hotfix #2 today (thin): forgot jars [analytics/refinery@f6d73c8] (duration: 00m 09s) |
[production] |
| 19:20 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@f6d73c8] (thin): Hotfix #2 today (thin): forgot jars [analytics/refinery@f6d73c8] |
[production] |
| 19:17 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@f6d73c8]: Hotfix #2 today: forgot jars [analytics/refinery@f6d73c8] (duration: 16m 54s) |
[production] |
| 19:14 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.34 refs T253022 |
[production] |
| 19:01 |
<shdubsh> |
restart varnishmtail and atsmtail on cp5001.eqsin.wmnet |
[production] |
| 19:00 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@f6d73c8]: Hotfix #2 today: forgot jars [analytics/refinery@f6d73c8] |
[production] |
| 17:03 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.34 refs T253022 (duration: 01m 06s) |
[production] |
| 17:02 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.34 refs T253022 |
[production] |
| 16:32 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.34/extensions/Wikibase: T253804 Use ThrowingEntityTermStoreWriter when writers shouldn't be called (duration: 01m 15s) |
[production] |
| 15:37 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182] (duration: 00m 10s) |
[production] |
| 15:37 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182] |
[production] |
| 15:05 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182] (duration: 25m 59s) |
[production] |
| 15:02 |
<moritzm> |
installing exim4 security updates on jessie (stretch/buster already fixed) |
[production] |
| 14:39 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182] |
[production] |
| 14:33 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:30 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:01 |
<ema> |
atskafka 0.8 uploaded to buster-wikimedia T253551 |
[production] |
| 13:49 |
<godog> |
roll-restart prometheus k8s-staging to enable thanos upload - T252186 |
[production] |