2301-2350 of 10000 results (54ms)
2022-03-02 ยง
21:44 <mutante> rolling out scap 4.4.2 on 'all' T302919 [production]
21:36 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host dumpsdata1007.eqiad.wmnet with OS bullseye [production]
21:19 <dancy@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:767580|wmf-config: Undeploy the fawiki test survey from production (T300291)]] (duration: 00m 50s) [production]
21:13 <robh@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dumpsdata1007.eqiad.wmnet with OS bullseye [production]
21:10 <dancy@deploy1002> rebuilt and synchronized wikiversions files: testing scap 4.4.2 [production]
21:05 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host dumpsdata1007.eqiad.wmnet with OS bullseye [production]
21:00 <mutante> deploy1002 - upgraded scap to 4.4.2-1 T302919 [production]
20:48 <mutante> running test-deploy to devcluster (restbase) to test new scap version, succesful and then rolled back, as the docs say T302919 [production]
20:48 <dzahn@deploy1002> Finished deploy [restbase/deploy@0848b15] (dev-cluster): (no justification provided) (duration: 00m 41s) [production]
20:47 <dzahn@deploy1002> Started deploy [restbase/deploy@0848b15] (dev-cluster): (no justification provided) [production]
20:44 <mutante> testec 'scap pull' still worked on mwdebug1001; rolling out scap 4.4.2 to A:restbase-canary (T302919) [production]
20:38 <mutante> rolling out scap 4.4.2 to A:mw-canary or A:parsoid-canary or A:mw-jobrunner-canary (T302919) [production]
20:20 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dumpsdata1007.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:11 <robh@cumin1001> START - Cookbook sre.hosts.provision for host dumpsdata1007.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:07 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:03 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
19:57 <brennen@deploy1002> rebuilt and synchronized wikiversions files: (no justification provided) [production]
19:53 <brennen@deploy1002> rebuilt and synchronized wikiversions files: (no justification provided) [production]
19:47 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:46 <brennen@deploy1002> Synchronized php-1.38.0-wmf.24/extensions/ApiFeatureUsage: Backport: [[gerrit:767103|Add a non-namespaced alias for ApiFeatureUsageQueryEngineElastica (T302907)]] (duration: 00m 50s) [production]
19:45 <robh@cumin1001> START - Cookbook sre.hosts.provision for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:36 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:33 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
19:30 <mutante> stopped icinga-wm [production]
19:14 <brennen@deploy1002> Synchronized php: group1 wikis to 1.38.0-wmf.24 refs T300200 (duration: 00m 50s) [production]
19:13 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.24 refs T300200 [production]
19:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T300992)', diff saved to https://phabricator.wikimedia.org/P21729 and previous config saved to /var/cache/conftool/dbconfig/20220302-191323-ladsgroup.json [production]
19:10 <brennen> 1.38.0-wmf.24 train (T300200): no current blockers; proceeding to group1 [production]
18:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P21728 and previous config saved to /var/cache/conftool/dbconfig/20220302-185819-ladsgroup.json [production]
18:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P21727 and previous config saved to /var/cache/conftool/dbconfig/20220302-184314-ladsgroup.json [production]
18:30 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T300992)', diff saved to https://phabricator.wikimedia.org/P21726 and previous config saved to /var/cache/conftool/dbconfig/20220302-182809-ladsgroup.json [production]
18:26 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
18:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1148 (T300992)', diff saved to https://phabricator.wikimedia.org/P21725 and previous config saved to /var/cache/conftool/dbconfig/20220302-182153-ladsgroup.json [production]
18:21 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [production]
18:21 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [production]
18:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T300992)', diff saved to https://phabricator.wikimedia.org/P21724 and previous config saved to /var/cache/conftool/dbconfig/20220302-182145-ladsgroup.json [production]
18:14 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/shellbox-media: apply [production]
18:14 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/shellbox-media: apply [production]
18:14 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
18:13 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
18:13 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/proton: apply [production]
18:13 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/proton: apply [production]
18:13 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/mathoid: apply [production]
18:12 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/mathoid: apply [production]
18:12 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply [production]
18:12 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/linkrecommendation: apply [production]
18:12 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply [production]
18:12 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/eventstreams-internal: apply [production]
18:12 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/eventstreams: apply [production]