|
2020-07-18
§
|
| 21:41 |
<shdubsh> |
restart logstash on logstash200[456] |
[production] |
| 21:14 |
<shdubsh> |
bounce logstash on logstash1007 |
[production] |
| 21:10 |
<shdubsh> |
bounce logstash on logstash1008 |
[production] |
| 21:06 |
<shdubsh> |
bounce logstash on logstash1009 |
[production] |
| 20:52 |
<marostegui> |
Due to db1082 crash there will be replication lag on s5 on labsdb hosts - T258336 |
[production] |
| 20:37 |
<cdanis@cumin1001> |
dbctl commit (dc=all): 'depool db1082, it crashed', diff saved to https://phabricator.wikimedia.org/P11951 and previous config saved to /var/cache/conftool/dbconfig/20200718-203704-cdanis.json |
[production] |
| 00:13 |
<dpifke> |
Performing one-time expiration of ArcLamp files older than 40 days (normal retention is 45 days), to solve disk space issue until either Ganeti issue is solved or compressed logfile support is merged. |
[production] |
|
2020-07-17
§
|
| 21:16 |
<dpifke> |
Removing MongoDB packages and data from webperf1002. |
[production] |
| 17:39 |
<dpifke@deploy1001> |
Finished deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) (duration: 00m 05s) |
[production] |
| 17:38 |
<dpifke@deploy1001> |
Started deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) |
[production] |
| 13:53 |
<akosiaris> |
powercycle kubernetes2002 |
[production] |
| 12:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P11944 and previous config saved to /var/cache/conftool/dbconfig/20200717-122400-marostegui.json |
[production] |
| 12:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11941 and previous config saved to /var/cache/conftool/dbconfig/20200717-120126-marostegui.json |
[production] |
| 11:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11940 and previous config saved to /var/cache/conftool/dbconfig/20200717-115155-marostegui.json |
[production] |
| 11:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11939 and previous config saved to /var/cache/conftool/dbconfig/20200717-113800-marostegui.json |
[production] |
| 11:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11938 and previous config saved to /var/cache/conftool/dbconfig/20200717-113050-marostegui.json |
[production] |
| 11:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1104', diff saved to https://phabricator.wikimedia.org/P11937 and previous config saved to /var/cache/conftool/dbconfig/20200717-112413-marostegui.json |
[production] |
| 09:15 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw1280.eqiad.wmnet |
[production] |
| 09:12 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: name=mw1280.eqiad.wmnet |
[production] |
| 08:48 |
<moritzm> |
imported prometheus-atlas-exporter 1.0+git20191204.ffafab7-2 to buster-wikimedia T247967 |
[production] |
| 08:29 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 08:05 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 07:54 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11936 and previous config saved to /var/cache/conftool/dbconfig/20200717-075124-marostegui.json |
[production] |
| 07:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1111', diff saved to https://phabricator.wikimedia.org/P11935 and previous config saved to /var/cache/conftool/dbconfig/20200717-074335-marostegui.json |
[production] |
| 07:34 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 07:34 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 07:33 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 07:33 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 07:32 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 07:30 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 06:30 |
<XioNoX> |
rename msw1-codfw interface range |
[production] |
| 06:28 |
<XioNoX> |
rename msw1-eqiad interface range |
[production] |
| 04:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1111', diff saved to https://phabricator.wikimedia.org/P11934 and previous config saved to /var/cache/conftool/dbconfig/20200717-044748-marostegui.json |
[production] |
| 04:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1092', diff saved to https://phabricator.wikimedia.org/P11933 and previous config saved to /var/cache/conftool/dbconfig/20200717-044658-marostegui.json |
[production] |
|
2020-07-16
§
|
| 22:15 |
<mutante> |
testreduce1001 manually git clone 'scandium' branch of integration/visualdiff into /srv/visualdiff (T257906) |
[production] |
| 21:54 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 3 (duration: 01m 49s) |
[production] |
| 21:52 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 3 |
[production] |
| 21:42 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 2 (duration: 01m 33s) |
[production] |
| 21:41 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 2 |
[production] |
| 21:40 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 (duration: 01m 01s) |
[production] |
| 21:39 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 |
[production] |
| 21:08 |
<cstone> |
payments-wiki revision changed from 91852dbc9b to bf91f8adff |
[production] |
| 20:32 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable client error logging on Catalan Wikipedia (T258073) (duration: 00m 57s) |
[production] |
| 19:32 |
<sbassett> |
Deployed mitigations for T257687 |
[production] |
| 19:14 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T248418 TimedMediaHandler: Make videojs the only player on all group0 (duration: 00m 57s) |
[production] |
| 18:54 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 18:53 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 18:50 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 18:49 |
<addshore> |
deployment windows finished with |
[production] |