701-750 of 10000 results (25ms)
2020-05-25 §
05:11 <marostegui> Deploy schema change on s6, directly on the master - T253342 [production]
04:54 <marostegui> Depool labsdb1011 - T249188 [production]
04:11 <kart_> Updated cxserver to 2020-05-22-083137-production (T246317, T252871) [production]
04:07 <kartik@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
04:04 <kartik@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
04:02 <kartik@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
2020-05-24 §
17:36 <gehel> restarting elasticsearch psi on elastic1052 [production]
16:44 <gehel> depool wdqs1007 to catch on lag [production]
16:43 <gehel> restart blazegraph on wdqs1007 [production]
13:51 <wm-bot> <lucaswerkmeister> deployed 5c6d1c6e30 (update Breton) [tools.lexeme-forms]
10:04 <elukey> re-run virtualpageview-hourly 23/05T15 - failed due to a sporadic kerberos/hive issue [analytics]
2020-05-23 §
23:41 <Krenair> Manually removed /etc/apt/sources.list.d/openstack-mitaka-jessie.list on utrs-database2 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/585814/1/modules/openstack/manifests/commonpackages/vms/mitaka.pp (missing absent) [utrs]
19:04 <krinkle@deploy1001> Synchronized php-1.35.0-wmf.31/includes/filerepo/file/LocalFile.php: I0f7e885997d60 (duration: 01m 06s) [production]
18:58 <krinkle@deploy1001> Synchronized php-1.35.0-wmf.32/includes/filerepo/file/LocalFile.php: I0f7e885997d60 (duration: 01m 08s) [production]
18:06 <krinkle@deploy1001> Synchronized php-1.35.0-wmf.32/includes/filerepo/: I31a9bb6672 (duration: 01m 06s) [production]
18:05 <krinkle@deploy1001> Synchronized php-1.35.0-wmf.31/includes/filerepo/: I31a9bb6672 (duration: 01m 10s) [production]
17:45 <Jayprakash12345> Migrate indic-wscontest to new toolforge URL (T253456) [tools.indic-wscontest]
15:44 <krinkle@deploy1001> Synchronized wmf-config/mc.php: I5ad8fe96b9098a8 - Disable coalesceKeys on commonswiki (duration: 01m 09s) [production]
14:58 <Krinkle> scap-pull to reset state on mwdebug1002 [production]
14:50 <Krinkle> Testing mc.php changes on mwdebug1002 [production]
08:04 <elukey> powercycle an-presto1004 - unresponsive, racadm getsel shows CPU overheating alerts [production]
05:04 <Operator873> restarted CVNBot15 [cvn]
01:20 <wm-bot> <peterbowman> Deploy 465b0a8: support new and legacy Toolforge URL schemes [tools.pbbot]
2020-05-22 §
22:42 <krinkle@deploy1001> Synchronized php-1.35.0-wmf.31/includes/filerepo/: Ie19613ef7643a (duration: 01m 06s) [production]
22:40 <krinkle@deploy1001> Synchronized php-1.35.0-wmf.32/includes/filerepo/: Ie19613ef7643a (duration: 01m 08s) [production]
21:58 <rxy> sudo kill $CVNBot6_pid [cvn]
21:21 <Jayprakash12345> Migrate wikifile-transfer to new toolforge URL (T253404) [tools.wikifile-transfer]
20:00 <bstorm_> rebooted tools-sgebastion-07 to clear up tmp file problems with 10 min warning [tools]
19:27 <bstorm_> removing /tmp files that begin with '-' because shell commands are failing [tools.image-metadata-viewer]
19:22 <bstorm_> removing /tmp/-mnt-nfs-labstore-secondary-tools-project-fvcbot-FPCBot-fpc-.lock because the filename is causing issues with shell commands [tools.fvcbot]
19:12 <bstorm_> running command to delete over 2000 tmp ca certs on tools-bastion-07 T253412 [tools]
17:43 <Examknow> restart bot [tools.mhwikibot]
15:58 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:58 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:57 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:53 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:47 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:45 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:45 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:30 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:25 <cdanis> fixing prometheus-nic-firmware-textfile.service wherever it is broken T253374 [production]
15:25 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
15:24 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
15:06 <marostegui> Decrease tendril_purge_global_status_log_5m storing rows time from 2 days to 1 day T252331 [production]
15:01 <kormat@cumin1001> dbctl commit (dc=all): 'Pool db2137 into s4+s5 T252985', diff saved to https://phabricator.wikimedia.org/P11292 and previous config saved to /var/cache/conftool/dbconfig/20200522-150120-kormat.json [production]
14:53 <reedy@deploy1001> Synchronized php-1.35.0-wmf.31/maintenance/blockUsers.php: (no justification provided) (duration: 01m 08s) [production]
14:51 <reedy@deploy1001> Synchronized php-1.35.0-wmf.32/maintenance/blockUsers.php: (no justification provided) (duration: 01m 09s) [production]
14:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1144:331[45] and db1097:331[45]', diff saved to https://phabricator.wikimedia.org/P11290 and previous config saved to /var/cache/conftool/dbconfig/20200522-143541-marostegui.json [production]
14:24 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
14:22 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]