2251-2300 of 10000 results (36ms)
2022-01-03 §
08:49 <moritzm> installing libpcap security updates [production]
08:36 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance [production]
08:36 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance [production]
08:32 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:28 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
08:25 <moritzm> installing zziplib security updates [production]
08:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2118.codfw.wmnet with reason: Maintenance [production]
08:07 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2118.codfw.wmnet with reason: Maintenance [production]
08:04 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance [production]
08:04 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance [production]
07:51 <moritzm> draining primary and secondary instances off ganeti2023 [production]
07:46 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
07:46 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
07:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
07:31 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
07:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2087.codfw.wmnet with reason: Maintenance [production]
07:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2087.codfw.wmnet with reason: Maintenance [production]
07:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2086.codfw.wmnet with reason: Maintenance [production]
07:23 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2086.codfw.wmnet with reason: Maintenance [production]
07:17 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
07:13 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
07:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
07:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
07:04 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
07:03 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
07:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
07:02 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:750831|Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708)]], Part I (duration: 01m 20s) [production]
07:01 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
07:00 <ladsgroup@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:750831|Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708)]], Part I (duration: 00m 58s) [production]
06:53 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance [production]
06:53 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance [production]
04:21 <Amir1> start of running populating actor in revision table on rest of sections. It will take two months to finish (T275246) [production]
2021-12-29 §
10:30 <elukey> kill tcpdump process on kubestagemaster1001 (kept a big pcap file opened that kept growing) [production]
2021-12-28 §
11:27 <godog> powercycle ms-be1059 -- was powered down [production]
02:39 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:35 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
02:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
02:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:17 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
02:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
02:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
2021-12-24 §
20:08 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@e282d2d]: (no justification provided) (duration: 00m 06s) [production]
20:08 <mforns@deploy1002> Started deploy [airflow-dags/analytics@e282d2d]: (no justification provided) [production]
04:20 <legoktm> depooled cp2029 now that it's up [production]
04:16 <legoktm> powercycling cp2029 via mgmt [production]
00:57 <ejegg> updated fundraising CiviCRM from 47dd67f2 to aaceb4ab [production]
00:28 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1003.eqiad.wmnet with OS buster [production]
00:28 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1004.eqiad.wmnet with OS buster [production]
00:28 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster [production]