Google Cloud Filestore in status REPAIRING blocks *everything* - google-cloud-platform

Question

We are using Google's Filestore cloud service for sharing files between our GCE VMs. Randomly, all processes seem to hang, especially interactive SSH sessions, and after some investigation we have determined that our Filestore, universally mounted across all VMs, was being repaired and was blocking all processes that tried to get any information on it.
I was able to log in as root and investigate, I noticed that all my interactive activity would hang, and eventually I pinpointed it to trying to stat the mountpoint of the Filestore instance. An strace df would hang like this:
statfs("/sys/kernel/config", {f_type=0x62656570, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
stat("/sys/kernel/config", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
statfs("/sys/fs/selinux", {f_type=SELINUX_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
stat("/sys/fs/selinux", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
statfs("/proc/sys/fs/binfmt_misc", {f_type=BINFMTFS_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
stat("/proc/sys/fs/binfmt_misc", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
statfs("/dev/hugepages", {f_type=HUGETLBFS_MAGIC, f_bsize=2097152, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=2097152, f_flags=ST_VALID|ST_RELATIME}) = 0
stat("/dev/hugepages", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
statfs("/mnt/local-storage", {f_type=0x58465342, f_bsize=4096, f_blocks=131007745, f_bfree=86129973, f_bavail=86129973, f_files=262143488, f_ffree=262141571, f_fsid={2065, 0}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
stat("/mnt/local-extra", {st_mode=S_IFDIR|0755, st_size=75, ...}) = 0
statfs("/mnt/shared-storage" ***HANG***
There was apparently no remedy except for waiting for the repair operation to complete. gcloud filestore operations list was showing no operations ongoing during that time. But gcloud filestore instances list would show the REPAIRING state like this:
[root#vm ~]# gcloud filestore instances list
INSTANCE_NAME ZONE TIER CAPACITY_GB FILE_SHARE_NAME IP_ADDRESS STATE CREATE_TIME
shared-storage europe-west1-b STANDARD 1024 shared_storage **.**.**.** REPAIRING 2019-08-09T16:03:02
Google Cloud Status Dashboard never showed any issue at or around the time.
Does anybody know why this happens and how to prevent it from happening, if possible. As shown in the output above, we are using the standard tier of Filestore.

Answer 1

We've configured coredumps to be written to the share from two dozen VMs, when a mass-death of our processes occurs, it seems that we reach the throughput limit of the share (standard tier) and that causes the share to enter the REPAIRING state, in turn blocking everything that tries to access it.
If you have a similar problem: check if it's possible that somehow you're reaching the throughput limit on your share.