|
2021-10-21
ยง
|
| 14:19 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 14:19 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 14:05 |
<ottomata> |
rerun refine_eventlogging_analytics refine_eventlogging_legacy and refine_event with -ignore-done-flag=true --since=2021-10-21T01:00:00 --until=2021-10-21T04:00:00 for backfill of missing data after gobblin problems |
[analytics] |
| 13:56 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 13:55 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 13:49 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 13:49 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 13:39 |
<btullis> |
btullis@an-launcher1002:~$ sudo systemctl restart gobblin-event_default |
[analytics] |
| 13:34 |
<volans> |
uploaded spicerack_1.0.6 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia |
[production] |
| 13:08 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:05 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:04 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.5 refs T281169 |
[production] |
| 12:58 |
<mdipietro> |
upgraded to 923250f7cd0b522259abdad450fbd2dfb16357bf which was really not an upgrade as the diff gave nothing. Though now it is clear what is deployed. |
[paws] |
| 12:56 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 7 hosts with reason: Schema change s3 T278619 |
[production] |
| 12:56 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 7 hosts with reason: Schema change s3 T278619 |
[production] |
| 12:52 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 14 hosts with reason: Schema change s1 T278619 |
[production] |
| 12:52 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 14 hosts with reason: Schema change s1 T278619 |
[production] |
| 12:48 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Schema change s4 T278619 |
[production] |
| 12:48 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 13 hosts with reason: Schema change s4 T278619 |
[production] |
| 12:43 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Schema change s2 T278619 |
[production] |
| 12:43 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 9 hosts with reason: Schema change s2 T278619 |
[production] |
| 12:34 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 11 hosts with reason: Schema change s7 T278619 |
[production] |
| 12:34 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 11 hosts with reason: Schema change s7 T278619 |
[production] |
| 11:55 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Schema change s5 T278619 |
[production] |
| 11:54 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 9 hosts with reason: Schema change s5 T278619 |
[production] |
| 11:47 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Schema change s6 T278619 |
[production] |
| 11:47 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 9 hosts with reason: Schema change s6 T278619 |
[production] |
| 11:35 |
<arturo> |
create project with btullis & elukey as projectadmins, quota 6 instances 12 cores and 18G ram (T292563) |
[data-engineering] |
| 11:13 |
<Lucas_WMDE> |
UTC morning backport+config window done |
[production] |
| 11:10 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/ResubmitChanges.php wikidatawiki --minimum-age $((60*60*12)) # T294008 |
[production] |
| 11:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:07 |
<jgiannelos@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:730848|Configure event stream for map tiles state change (T289771)]] (duration: 01m 04s) |
[production] |
| 11:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:48 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 10:48 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 10:48 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 10:47 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 10:35 |
<joal> |
Re-refine netflow data after gobblin pulled data fix |
[analytics] |
| 10:19 |
<arturo> |
drop firewall exception on core routers for wiki replicas legacy setup (T293897) |
[admin] |
| 10:14 |
<jbond> |
mergeing refactor of P:base Gerrit:714975 |
[production] |
| 10:12 |
<arturo> |
drop NAT exception for wiki replicas legacy setup (T293897) |
[admin] |
| 09:54 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:49 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:49 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/Wikibase/repo/maintenance/ResubmitChanges.php wikidatawiki # test T292728 |
[releng] |
| 09:48 |
<majavah> |
deploying toolforge-webservice 0.79 |
[tools] |
| 09:23 |
<arturo> |
bump quotas to instances 12 cores 82 ram 249856 gigabytes 600 volumes 10 (T293832) |
[gitlab-runners] |
| 08:56 |
<urbanecm@deploy1002> |
Synchronized private/PrivateSettings.php: Update T250887 mitigations (duration: 01m 03s) |
[production] |
| 08:41 |
<joal> |
Rerun webrequest-load jobs for hour 2021-10-21T02:00 |
[analytics] |
| 08:33 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp3062.esams.wmnet,service=(varnish-fe|ats-tls) |
[production] |
| 08:26 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp3062.esams.wmnet,service=(varnish-fe|ats-tls) |
[production] |