|
2020-05-28
ยง
|
| 21:19 |
<bd808> |
Killed 7 python processes run by user 'mattho69' on login.toolforge.org |
[tools] |
| 21:12 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.34/includes/specials/SpecialUserrights.php: T253909 Restore visibility (previously implicitely public) (duration: 01m 06s) |
[production] |
| 21:06 |
<bstorm_> |
upgrading tools-k8s-worker-[30-60] to kubernetes 1.16.10 T246122 |
[tools] |
| 20:38 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.32/skins/Vector/resources/skins.vector.styles: T253905 HOTFIX: Do not apply p-personal absolute positioning to all menus (duration: 01m 07s) |
[production] |
| 20:22 |
<shdubsh> |
restart varnishmtail and atsmtail eqsin |
[production] |
| 20:11 |
<shdubsh> |
restart ncredirmtail on ncredir5001 |
[production] |
| 19:20 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: roll back the train due to T253905 |
[production] |
| 19:20 |
<twentyafterfour> |
group2 back to wmf.32 due to T253905 |
[production] |
| 19:20 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@f6d73c8] (thin): Hotfix #2 today (thin): forgot jars [analytics/refinery@f6d73c8] (duration: 00m 09s) |
[production] |
| 19:20 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@f6d73c8] (thin): Hotfix #2 today (thin): forgot jars [analytics/refinery@f6d73c8] |
[production] |
| 19:17 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@f6d73c8]: Hotfix #2 today: forgot jars [analytics/refinery@f6d73c8] (duration: 16m 54s) |
[production] |
| 19:14 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.34 refs T253022 |
[production] |
| 19:01 |
<shdubsh> |
restart varnishmtail and atsmtail on cp5001.eqsin.wmnet |
[production] |
| 19:00 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@f6d73c8]: Hotfix #2 today: forgot jars [analytics/refinery@f6d73c8] |
[production] |
| 18:31 |
<milimetric> |
after deployment, restarted four oozie jobs with new SLAs and fixed datasets definitions |
[analytics] |
| 17:54 |
<bstorm_> |
upgraded tools-k8s-worker-[11..15] and starting on -21-29 now T246122 |
[tools] |
| 17:52 |
<James_F> |
Zuul: [integration/docroot] Switch from php56 to php70 |
[releng] |
| 17:03 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.34 refs T253022 (duration: 01m 06s) |
[production] |
| 17:02 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.34 refs T253022 |
[production] |
| 16:32 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.34/extensions/Wikibase: T253804 Use ThrowingEntityTermStoreWriter when writers shouldn't be called (duration: 01m 15s) |
[production] |
| 16:10 |
<James_F> |
Zuul: Move MediaModeration to production section and enforce T247943 |
[releng] |
| 16:01 |
<bstorm_> |
kubectl upgraded to 1.16.10 on all bastions T246122 |
[tools] |
| 15:58 |
<arturo> |
upgrading tools-k8s-worker-[1..10] to 1.16.10 (T246122) |
[tools] |
| 15:41 |
<arturo> |
upgrading tools-k8s-control-3 to 1.16.10 (T246122) |
[tools] |
| 15:37 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182] (duration: 00m 10s) |
[production] |
| 15:37 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182] |
[production] |
| 15:17 |
<arturo> |
upgrading tools-k8s-control-2 to 1.16.10 (T246122) |
[tools] |
| 15:09 |
<arturo> |
upgrading tools-k8s-control-1 to 1.16.10 (T246122) |
[tools] |
| 15:05 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182] (duration: 25m 59s) |
[production] |
| 15:02 |
<moritzm> |
installing exim4 security updates on jessie (stretch/buster already fixed) |
[production] |
| 14:49 |
<arturo> |
cleanup /etc/apt/sources.list.d/ directory in all tools-k8s-* VMs |
[tools] |
| 14:39 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182] |
[production] |
| 14:33 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:30 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:26 |
<ZI_Jony> |
restarted CVNBot14 |
[cvn] |
| 14:01 |
<ema> |
atskafka 0.8 uploaded to buster-wikimedia T253551 |
[production] |
| 13:49 |
<godog> |
roll-restart prometheus k8s-staging to enable thanos upload - T252186 |
[production] |
| 13:36 |
<hashar> |
Restarting CI Jenkins for plugin rollback |
[production] |
| 13:36 |
<andrewbogott> |
rebuilding cloudservices2002-dev with Buster |
[admin] |
| 13:31 |
<hashar> |
Rolling back Xunit Jenkins plugin # T194096 |
[releng] |
| 11:49 |
<moritzm> |
installing unbound security updates |
[production] |
| 11:27 |
<arturo> |
cleanup livehackings |
[toolsbeta] |
| 11:27 |
<arturo> |
merging change to front-proxy: https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 (T253816) |
[tools] |
| 11:03 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Add db2138 to s2+s4 T252985', diff saved to https://phabricator.wikimedia.org/P11330 and previous config saved to /var/cache/conftool/dbconfig/20200528-110333-kormat.json |
[production] |
| 10:36 |
<jayme@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 10:34 |
<jayme@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 10:31 |
<arturo> |
livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 (T253816) |
[toolsbeta] |
| 10:30 |
<arturo> |
livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 |
[toolsbeta] |
| 10:30 |
<jayme@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 10:21 |
<RhinosF1> |
Known inconsistency in git - unable to pull changes (NOW RESOLVED) |
[tools.zppixbot] |