|
2020-10-05
ยง
|
| 20:28 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:26 |
<ebernhardson> |
restart elasticsearch_6@production-search-codfw on elastic2051 to take reduced (32 sector, 16kB) readahead settings T264053 |
[production] |
| 20:13 |
<ebernhardson> |
restart elasticsearch_6@production-search-codfw on elastic2051 to take reduced (64 sector, 32kB) readahead settings T264053 |
[production] |
| 19:56 |
<ebernhardson> |
restart elasticsearch_6@production-search-codfw on elastic2050 to take reduced (128kB) readahead settings T264053 |
[production] |
| 19:31 |
<mutante> |
ran sre.dns.netbox to push addition of an-worker1113 which was commited in prod repo but not in netbox data |
[production] |
| 19:30 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:27 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:59 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@2c6c335] (thin): [THIN] Special deployment to unblock deletion jobs [analytics/refinery@2c6c335e61cecd0321ec6f066a153feaf2dbbc27] (duration: 00m 08s) |
[production] |
| 18:59 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@2c6c335] (thin): [THIN] Special deployment to unblock deletion jobs [analytics/refinery@2c6c335e61cecd0321ec6f066a153feaf2dbbc27] |
[production] |
| 18:58 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@2c6c335]: Special deployment to unblock deletion jobs [analytics/refinery@2c6c335e61cecd0321ec6f066a153feaf2dbbc27] (duration: 12m 08s) |
[production] |
| 18:46 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@2c6c335]: Special deployment to unblock deletion jobs [analytics/refinery@2c6c335e61cecd0321ec6f066a153feaf2dbbc27] |
[production] |
| 18:17 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) |
[production] |
| 18:17 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers |
[production] |
| 18:15 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) |
[production] |
| 18:13 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers |
[production] |
| 18:11 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) |
[production] |
| 18:10 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers |
[production] |
| 17:53 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:51 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:29 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:27 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:25 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . |
[production] |
| 17:25 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 17:00 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 17:00 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . |
[production] |
| 16:59 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . |
[production] |
| 16:59 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 16:51 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 16:51 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . |
[production] |
| 15:15 |
<ppchelko@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . |
[production] |
| 14:56 |
<ppchelko@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . |
[production] |
| 14:55 |
<ppchelko@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 14:41 |
<elukey> |
shutdown stat1005 and stat1008 for ram expansion (1005 again) |
[production] |
| 14:36 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@366a543]: T263133 T264035 (duration: 22m 23s) |
[production] |
| 14:25 |
<elukey> |
shutdown an-master1001 for ram expansion |
[production] |
| 14:13 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@366a543]: T263133 T264035 |
[production] |
| 14:01 |
<filippo@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 13:58 |
<filippo@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 13:55 |
<filippo@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . |
[production] |
| 13:54 |
<elukey> |
shutdown stat1005 for ram upgrade |
[production] |
| 13:31 |
<elukey> |
shutdown an-master1002 for ram expansion (64 -> 128G) |
[production] |
| 12:39 |
<moritzm> |
installing curl security updates on remaining hosts |
[production] |
| 11:34 |
<hoo@deploy1001> |
Synchronized wmf-config/: Revert "Remove $wgExtraLanguageNames from Wikidata and Commons" (T264295) (duration: 00m 59s) |
[production] |
| 11:28 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: be73f155001e9095697c3c21a208c63e7bf5d2d1: Move changetags right from users to sysop [trwiki] (T264508) (duration: 00m 59s) |
[production] |
| 11:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: cd30b626e23b48146b970c72731f8f7bb1eee9e1: wgSkipSkins: Exclude contenttranslation skin from skin options for users (T263093) (duration: 00m 59s) |
[production] |
| 11:05 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:632212| Bumping portals to master (T128546)]] (duration: 00m 58s) |
[production] |
| 11:04 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:632212| Bumping portals to master (T128546)]] (duration: 00m 58s) |
[production] |
| 10:37 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 10:37 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:632204| Bumping portals to master (T128546)]] (duration: 00m 58s) |
[production] |
| 10:36 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:632204| Bumping portals to master (T128546)]] (duration: 01m 00s) |
[production] |