3101-3150 of 10000 results (25ms)
2020-07-27 §
06:44 <elukey> truncate big log file on an-launcher1002 that is filling up the /srv partition [production]
06:36 <elukey> apt-get clean on netbox1001 to free some space [production]
05:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1138 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12043 and previous config saved to /var/cache/conftool/dbconfig/20200727-051156-marostegui.json [production]
05:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2087:3316, db2087:3317 for on-site maintenance T258587', diff saved to https://phabricator.wikimedia.org/P12042 and previous config saved to /var/cache/conftool/dbconfig/20200727-050058-marostegui.json [production]
04:58 <marostegui> Stop MySQL on db2087 for on-site maintenance T258587 [production]
2020-07-25 §
12:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1096:3315 into s5 api afte db1082 crashed T258336', diff saved to https://phabricator.wikimedia.org/P12041 and previous config saved to /var/cache/conftool/dbconfig/20200725-124104-marostegui.json [production]
09:16 <oblivian@cumin1001> dbctl commit (dc=all): 'Depool db1082 T258336', diff saved to https://phabricator.wikimedia.org/P12040 and previous config saved to /var/cache/conftool/dbconfig/20200725-091616-oblivian.json [production]
01:52 <mutante> ganeti - also removing (unmounted) disk 2 (100G) from webperf1002. T257931 [production]
00:46 <mutante> ganeti - removing disk 3 (20G) from webperf1002. the disks are 0-indexed, so the ones actually mounted are 0 (50G) and 1 (300G) (T257931) [production]
00:42 <dpifke> Manually compressing some more data on webperf1002, using arclamp-compress-logs from https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/615904. [production]
2020-07-24 §
23:00 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
20:08 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:06 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:57 <dpifke> Manually gzipping some older ArcLamp data on webperf1002, to free up space and verify new compression support. [production]
19:55 <dpifke@deploy1001> Finished deploy [performance/arc-lamp@772b4a3]: Deploy CLs 611465 and 613740 to add compression support to ArcLamp (duration: 00m 05s) [production]
19:55 <dpifke@deploy1001> Started deploy [performance/arc-lamp@772b4a3]: Deploy CLs 611465 and 613740 to add compression support to ArcLamp [production]
16:55 <Amir1> deployment done [production]
16:49 <ladsgroup@deploy1001> Synchronized php-1.36.0-wmf.1/extensions/Wikibase/repo/includes/RepoHooks.php: [[gerrit:616032|Prevent onTitleGetRestrictionTypes changing ns0 protections]], Part II (duration: 01m 07s) [production]
16:47 <ladsgroup@deploy1001> Synchronized php-1.36.0-wmf.1/extensions/Wikibase/repo/includes/WikibaseRepo.php: [[gerrit:616032|Prevent onTitleGetRestrictionTypes changing ns0 protections]], Part I (duration: 01m 06s) [production]
15:06 <reedy@deploy1001> Finished scap: Score backports (duration: 36m 50s) [production]
14:30 <reedy@deploy1001> Started scap: Score backports [production]
13:31 <XioNoX> advertise 185.71.138.0/24 from AMS [production]
13:17 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
13:00 <ladsgroup@deploy1001> Synchronized php-1.36.0-wmf.1/includes/import/ImportableOldRevisionImporter.php: [[gerrit:616029|Import: use master DB for loading slots.]] (T258666) (duration: 01m 07s) [production]
12:34 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
12:04 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
11:48 <hnowlan> bootstrapped restbase-dev1004-b [production]
11:13 <hnowlan> started bootstrap of restbase-dev1004-a [production]
10:51 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:49 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:35 <hnowlan> started reimage of restbase-dev1004 [production]
09:59 <jmm@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
09:48 <jmm@cumin1001> START - Cookbook sre.ganeti.makevm [production]
08:40 <kormat> restarting mariadb on all sanitarium hosts T258711 [production]
08:35 <akosiaris> start nagios-nrpe-server on kubernetes2002 [production]
07:44 <elukey> depool wtp1025 - disk full [production]
06:30 <tstarling@deploy1001> Started scap: for Score [production]
02:36 <tstarling@deploy1001> Synchronized php-1.36.0-wmf.1/extensions/Score/includes/Score.php: removing superseded local patch for hard-coding lilypond version (duration: 01m 09s) [production]
01:19 <ejegg> updated payments-wiki from 31a3de1130 to c365c136d2 [production]
01:04 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
01:02 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
01:02 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
01:02 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
01:02 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
01:02 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
01:02 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
01:02 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
01:02 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
01:02 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
01:02 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]