|
2020-08-20
§
|
| 10:57 |
<XioNoX> |
re-pool codfw - T259621 |
[production] |
| 10:53 |
<XioNoX> |
un-drain cr1-codfw - T259621 |
[production] |
| 10:45 |
<XioNoX> |
cr1-codfw> request chassis routing-engine master switch - T259621 |
[production] |
| 10:26 |
<hashar> |
Restarted zuul-merger instances on contint1001 and contint2001 |
[production] |
| 10:24 |
<hashar@deploy1001> |
Finished deploy [zuul/deploy@8a05b4d]: Support Gerrit replication events (duration: 00m 24s) |
[production] |
| 10:24 |
<hashar@deploy1001> |
Started deploy [zuul/deploy@8a05b4d]: Support Gerrit replication events |
[production] |
| 10:21 |
<XioNoX> |
cr1-codfw> request chassis routing-engine master switch - T259621 |
[production] |
| 10:12 |
<XioNoX> |
reboot cr1-codfw:re1 (backup) for upgrade - T259621 |
[production] |
| 09:57 |
<XioNoX> |
bump cr1-codfw OSPF metrics - T259621 |
[production] |
| 09:51 |
<XioNoX> |
enable transit/peering and re-set normal OSPF values on cr2-codfw - T259621 |
[production] |
| 09:41 |
<XioNoX> |
cr2-codfw> request chassis routing-engine master switch - T259621 |
[production] |
| 09:36 |
<eileen> |
civicrm revision changed from cf9fadbeed to 6c9441a18e, config revision is 706cf3c898 |
[production] |
| 09:33 |
<XioNoX> |
reboot cr2-codfw:re0 (backup) for upgrade - T259621 |
[production] |
| 09:18 |
<XioNoX> |
cr2-codfw> request chassis routing-engine master switch - T259621 |
[production] |
| 09:18 |
<kormat> |
stress-testing db2125 T260670 |
[production] |
| 09:08 |
<XioNoX> |
reboot cr2-codfw:re1 (backup) for upgrade - T259621 |
[production] |
| 09:03 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Repool db2125 after host failure T260670', diff saved to https://phabricator.wikimedia.org/P12303 and previous config saved to /var/cache/conftool/dbconfig/20200820-090313-kormat.json |
[production] |
| 08:52 |
<kormat> |
removing /usr/bin/check_mariadb.py from all db hosts T259516 |
[production] |
| 08:52 |
<XioNoX> |
disable transit/peering on cr2-codfw - T259621 |
[production] |
| 08:48 |
<XioNoX> |
bump cr2-codfw OSPF metrics - T259621 |
[production] |
| 08:44 |
<jynus> |
running analyze table on db1115's tendril.global_status_log, may case some stalls on tendril/dbtree T260876 |
[production] |
| 08:41 |
<XioNoX> |
depool codfw for routers upgrade - T259621 |
[production] |
| 08:31 |
<XioNoX> |
enable transit/peering on cr3-knams - T259621 |
[production] |
| 08:21 |
<XioNoX> |
reboot cr3-knams for upgrade - T259621 |
[production] |
| 08:07 |
<XioNoX> |
disable transit/peering on cr3-knams - T259621 |
[production] |
| 07:39 |
<hashar> |
contint2001: restarted zuul |
[production] |
| 07:29 |
<hashar> |
contint1001: restarted zuul-merger |
[production] |
| 07:29 |
<hashar@deploy1001> |
Finished deploy [zuul/deploy@5989ed0]: Upgrade gear from 0.7.0 to 1.15.1+wmf1 - T258630 (duration: 00m 13s) |
[production] |
| 07:28 |
<hashar@deploy1001> |
Started deploy [zuul/deploy@5989ed0]: Upgrade gear from 0.7.0 to 1.15.1+wmf1 - T258630 |
[production] |
| 01:54 |
<ejegg> |
re-enabled fundraising scheduled jobs |
[production] |
| 00:51 |
<mutante> |
ms-be1039 - started failed ferm service |
[production] |
| 00:34 |
<ejegg> |
stopped fundraising scheduled jobs |
[production] |
| 00:27 |
<eileen> |
civicrm revision changed from c442a09153 to cf9fadbeed, config revision is 3cdffd4fc2 |
[production] |
|
2020-08-19
§
|
| 23:20 |
<Urbanecm> |
Evening B&C window closed |
[production] |
| 23:08 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: a80899948c26ca36b970b80fbad07600fe4ce92c: Enable VisualEditor in namespaces Draft and Wikiproject on hywiki (T260825) (duration: 01m 05s) |
[production] |
| 22:41 |
<eileen> |
civicrm revision changed from 34f95a3311 to c442a09153, config revision is 3cdffd4fc2 |
[production] |
| 21:27 |
<eileen> |
civicrm revision changed from 154519cc1f to 34f95a3311, config revision is 3cdffd4fc2 |
[production] |
| 21:17 |
<cdanis@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 21:17 |
<cdanis@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 20:39 |
<dpifke@deploy1001> |
Finished deploy [performance/arc-lamp@2ef1af7]: Deploy fixes for notifications and OOM prevention (T259167) (duration: 00m 06s) |
[production] |
| 20:39 |
<dpifke@deploy1001> |
Started deploy [performance/arc-lamp@2ef1af7]: Deploy fixes for notifications and OOM prevention (T259167) |
[production] |
| 19:43 |
<ebernhardson> |
restart mjolnir-kafka-bulk-daemon on search-loader2001 with debug logging |
[production] |
| 19:20 |
<mutante> |
testreduce1001 - re-enabled puppet, confirmed parsoid-rt service was now stopped properly by puppet while it runs as before on scandium, the previous parsoid-testing host. switching it over is now a Hiera one-liner. (T257906) |
[production] |
| 19:15 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.36.0-wmf.5 refs T257973 (duration: 01m 04s) |
[production] |
| 19:14 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.5 refs T257973 |
[production] |
| 19:02 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 60af096b80a8ef7bc94ec40ce203fd27b0c97f26: Add autopatrolled group at arzwiki (T260761) (duration: 01m 04s) |
[production] |
| 18:52 |
<mutante> |
testreduce1001 - disable puppet; stop parsoid-rt service |
[production] |
| 18:47 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 924a03bd624d6750a7e776e09713056cc45e5cc5: Add clinton.presidentiallibraries.us to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T259927) (duration: 01m 04s) |
[production] |
| 18:45 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: 83b34e1bd1ed804a70f67e089580e082f89e2a0f: ClosedWikiProvider: Use testUserForCreation rather than testForAuthentication (T258695) (duration: 01m 04s) |
[production] |
| 18:39 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 95d45f6e002df78d4860a711042d77a6b0bdecb9: Dont index Draft (118) and Draft talk (119) on hywiki (T260804) (duration: 01m 04s) |
[production] |