|
2018-07-01
§
|
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 v |
[production] |
| 02:54 |
<hjukhjuki> |
forcing reboot of cp3033 via mgmt |
[production] |
| 02:53 |
<hjukhjuki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 02:44 |
<andrewbogott> |
forcing reboot of labservices1001 via mgmt |
[production] |
|
2018-06-30
§
|
| 21:35 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 (duration: 02m 27s) |
[production] |
| 21:32 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 |
[production] |
| 20:40 |
<Krenair> |
ran git gc on deployment-tin:/srv/mediawiki to free up space |
[releng] |
| 20:40 |
<Krenair> |
ran git gc on deployment-tin:/srv/mediawiki to free up space |
[deployment-prep] |
| 18:15 |
<chicocvenancio> |
pushed new config to PAWS to fix dumps nfs mountpoint |
[tools] |
| 16:40 |
<zhuyifei1999_> |
because tools-paws-master-01 was having ~1000 loadavg due to NFS having issues and processes stuck in D state |
[tools] |
| 16:39 |
<zhuyifei1999_> |
reboot tools-paws-master-01 |
[tools] |
| 16:35 |
<zhuyifei1999_> |
`root@tools-paws-master-01:~# sed -i 's/^labstore1006.wikimedia.org/#labstore1006.wikimedia.org/' /etc/fstab` |
[tools] |
| 16:34 |
<andrewbogott> |
"sed -i '/labstore1006/d' /etc/fstab" everywhere |
[tools] |
| 16:28 |
<chasemp> |
labstore1006:~# service nfs-kernel-server restart |
[production] |
|
2018-06-29
§
|
| 23:49 |
<bd808> |
Stopped webservice due to CrashLoopBackOff failures. Caused by "ImportError: No module named 'yamlconf'" |
[tools.ores-support-checklist] |
| 23:44 |
<bd808> |
test |
[tools.stashbot] |
| 21:37 |
<chasemp> |
(late log) of VLAN creations for T184209 cloud-instances2-b-eqiad and cloud-instance-transport1-b-eqiad |
[production] |
| 19:11 |
<herron> |
stat1005:~# killall -u ironholds T197895 |
[production] |
| 17:41 |
<bd808> |
Rescheduling continuous jobs away from tools-exec-1408 where load is high |
[tools] |
| 17:28 |
<XioNoX> |
Re-activating v6 BGP session to Telia on cr2-eqiad - T198502 |
[production] |
| 17:21 |
<XioNoX> |
deactivating v6 BGP session to Telia on cr2-eqiad - T198502 |
[production] |
| 17:11 |
<bd808> |
Rescheduled jobs away from toole-exec-1404 where linkwatcher is currently stealing most of the CPU (T123121) |
[tools] |
| 16:46 |
<bd808> |
Killed orphan tool owned processes running on the job grid. Mostly jembot and wsexport php-cgi processes stuck in deadlock following an OOM. T182070 |
[tools] |
| 14:43 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 00m 50s) |
[production] |
| 12:30 |
<moritzm> |
uploaded ffmpeg 3.2.10-1~deb9u1+wmf1 to apt.wikimedia.org/stretch-wikimedia (component/vp9) (linked against libvpx 1.7 and with backported row-mt support) (T190333) |
[production] |
| 11:11 |
<moritzm> |
installing zsh updates from jessie 8.11 point release |
[production] |
| 11:01 |
<moritzm> |
installing lame security updates |
[production] |
| 10:34 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2058 after reimage (duration: 00m 51s) |
[production] |
| 10:21 |
<mobrovac@deploy1001> |
Finished deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853 (duration: 03m 26s) |
[production] |
| 10:20 |
<marostegui> |
Deploy schema change on db1092 T191316 T192926 T89737 T195193 |
[production] |
| 10:19 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 00m 50s) |
[production] |
| 10:18 |
<mobrovac@deploy1001> |
Started deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853 |
[production] |
| 10:14 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 00m 50s) |
[production] |
| 09:23 |
<marostegui> |
Stop MySQL on db2058 to reimage it |
[production] |
| 09:23 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2058 for reimage (duration: 00m 50s) |
[production] |
| 09:18 |
<moritzm> |
uploaded libvpx 1.7.0-3+wmf1 to apt.wikimedia.org/stretch-wikimedia (component/vp9) (T190333) |
[production] |
| 09:08 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2054 after reimage (duration: 00m 51s) |
[production] |
| 08:39 |
<vgutierrez> |
Apply new TLS varnishkafka settings in cache::text nodes - T182993 |
[production] |
| 08:27 |
<akosiaris> |
upgrade php on phab1001 to 5.6.33+dfsg-0+deb8u1 |
[production] |
| 08:03 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool es1016 (duration: 00m 50s) |
[production] |
| 07:53 |
<marostegui> |
Stop MySQL on db2054 for reimage |
[production] |
| 07:53 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2054 for reimage (duration: 00m 50s) |
[production] |
| 06:21 |
<jynus> |
stop and reimage es1016 |
[production] |
| 05:40 |
<elukey> |
force umount of dumps labstore nfs mountpoints on stat100[56]/notebook100[34] to reduce load (also too many open files) |
[production] |
| 05:02 |
<marostegui> |
Deploy schema change on db1104 T191316 T192926 T89737 T195193 |
[production] |
| 05:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 00m 50s) |
[production] |