|
2018-09-18
§
|
| 16:33 |
<mutante> |
mwmaint2001 - nagios-nrpe-server had status 'failed' and caused all NRPE Icinga checks to fail but recovered after simply starting it again |
[production] |
| 16:32 |
<mutante> |
radon - re-enabled disabled puppet without reason (decom) T202040 |
[production] |
| 16:32 |
<mutante> |
mwmaint2001 - starting nagios-nrpe-server |
[production] |
| 16:22 |
<XioNoX> |
delete `term cumin` from cr1/2-eqiad analytics filter (already permited by established-tcp term) |
[production] |
| 16:02 |
<moritzm> |
installing policykit-1 security updates on jessie |
[production] |
| 15:55 |
<elukey@deploy1001> |
Finished deploy [analytics/refinery@1a6235a]: Fix cron scrips from the NYC offsite (duration: 09m 32s) |
[production] |
| 15:53 |
<XioNoX> |
add cumin2001 to mr* security policies - T204730 |
[production] |
| 15:47 |
<moritzm> |
installing spice security updates |
[production] |
| 15:46 |
<elukey@deploy1001> |
Started deploy [analytics/refinery@1a6235a]: Fix cron scrips from the NYC offsite |
[production] |
| 15:26 |
<XioNoX> |
add cumin2001 to labs-in4 on cr1/2-eqiad |
[production] |
| 15:23 |
<hashar@deploy1001> |
Finished scap: Sync again testwiki to php-1.32.0-wmf.20, I might have screwed up l10ncache (duration: 06m 31s) |
[production] |
| 15:20 |
<hashar> |
hashar scap for testwiki is actually php-1.32.0-wmf.22 |
[production] |
| 15:19 |
<mepps> |
updated process-control to c03857f6a6 |
[production] |
| 15:17 |
<hashar@deploy1001> |
Started scap: Sync again testwiki to php-1.32.0-wmf.20, I might have screwed up l10ncache |
[production] |
| 15:15 |
<mepps> |
updated process-control to f9ab984c5d |
[production] |
| 15:07 |
<hashar@deploy1001> |
Finished scap: testwiki to php-1.32.0-wmf.20 (duration: 35m 46s) |
[production] |
| 14:31 |
<hashar@deploy1001> |
Started scap: testwiki to php-1.32.0-wmf.20 |
[production] |
| 14:19 |
<marostegui> |
Drop T153638_echo_XXX tables from db1123 |
[production] |
| 14:02 |
<marostegui> |
Drop T153638_echo_XXX tables from db1095:3313 - T153638 |
[production] |
| 13:58 |
<papaul> |
upgrading BIOS on ms-be2030 |
[production] |
| 13:57 |
<banyek@deploy1001> |
Synchronized wmf-config/db-eqiad.php: T203565: Repool db1114 and db1119 (duration: 00m 49s) |
[production] |
| 13:56 |
<godog> |
poweroff ms-be2030 - T204567 |
[production] |
| 13:48 |
<moritzm> |
updating intel-microcode on Debian jessie/stretch to 3.20180807a.1 |
[production] |
| 13:43 |
<moritzm> |
installing redis security updates on maps* servers |
[production] |
| 13:32 |
<godog> |
repair sdd on ms-be2041 - T199198 |
[production] |
| 13:31 |
<marostegui> |
Rename echo_XXX tables on db1095 and db1123 |
[production] |
| 13:26 |
<moritzm> |
reimaging mw2245 (spare host) to test reimages from cumin2001 |
[production] |
| 13:24 |
<marostegui> |
Drop T153638_echo_XXX tables from db1078 - T153638 |
[production] |
| 13:21 |
<hashar> |
Cutting branches wmf/1.32.0-wmf.22 | T191068 |
[production] |
| 13:10 |
<marostegui> |
Drop T153638_echo_XXX tables from db1077 - T153638 |
[production] |
| 12:12 |
<moritzm> |
rebooting labnodepool1001 for kernel security update |
[production] |
| 10:36 |
<jynus> |
stopping dbstore2002:s2 mariadb instance |
[production] |
| 10:31 |
<jynus> |
stop db2041 to clone to dbstore2002 |
[production] |
| 10:25 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2041 (duration: 00m 49s) |
[production] |
| 10:24 |
<marostegui> |
Rename echo_XXX tables on db1077 and db1078 - T153638 |
[production] |
| 10:12 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Tune the s3 and s5 database loads (duration: 00m 49s) |
[production] |
| 10:09 |
<marostegui> |
Drop T153638_echo_XXX tables from dbstore2002:3313 - T153638 |
[production] |
| 10:06 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1105 (duration: 00m 49s) |
[production] |
| 10:04 |
<hashar> |
ci: updating Quibble Jenkins jobs to 0.0.26 |
[production] |
| 10:01 |
<marostegui> |
Rename echo_XXX tables on dbstore2002:3313 - T153638 |
[production] |
| 09:37 |
<marostegui> |
Drop T153638_echo_XXX tables from db2057 - T153638 |
[production] |
| 09:10 |
<jynus> |
stop mariadb (both instances) and restart db1105 for upgrade |
[production] |
| 08:54 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully depool db1105 (duration: 00m 49s) |
[production] |
| 08:45 |
<banyek> |
stop replication & stop mysql on db1119 (preparing to clone db1114) |
[production] |
| 08:02 |
<godog> |
bounce rsyslog on wezen/lithium, tls listener was timing out in icinga |
[production] |
| 08:02 |
<jynus> |
stop and restart db1105 for upgrade |
[production] |
| 07:53 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1105 (duration: 00m 50s) |
[production] |
| 06:55 |
<moritzm> |
installin zsh security updates |
[production] |
| 06:21 |
<marostegui> |
Drop tmp_2 and tmp_3 index from wikidatawiki.recentchanges on dbstore2001, db2079, db2082,db2083 - T202764 |
[production] |
| 06:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool es1013 after kernel and mariadb upgrade (duration: 00m 49s) |
[production] |