mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-05-30 00:25:34 +00:00
Adding troubleshooting info to logging page - inotify system update
Signed-off-by: Sunil Singh <sunil.singh@suse.com>
This commit is contained in:
@@ -116,6 +116,19 @@ By default, Rancher collects logs for control plane components and node componen
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### The Logging Buffer Overloads Pods
|
||||
|
||||
Depending on your configuration, the default buffer size may be too large and cause pod failures. One way to reduce the load is to lower the logger's flush interval. This prevents logs from overfilling the buffer. You can also add more flush threads to handle moments when many logs are attempting to fill the buffer at once.
|
||||
|
||||
+13
@@ -79,6 +79,19 @@ Rancher Logging 有两个角色,分别是 `logging-admin` 和 `logging-view`
|
||||
|
||||
## 故障排除
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### 日志缓冲区导致 Pod 过载
|
||||
|
||||
根据你的配置,默认缓冲区大小可能太大并导致 Pod 故障。减少负载的一种方法是降低记录器的刷新间隔。这可以防止日志溢出缓冲区。你还可以添加更多刷新线程来处理大量日志试图同时填充缓冲区的情况。
|
||||
|
||||
+13
@@ -79,6 +79,19 @@ Rancher Logging 有两个角色,分别是 `logging-admin` 和 `logging-view`
|
||||
|
||||
## 故障排除
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### 日志缓冲区导致 Pod 过载
|
||||
|
||||
根据你的配置,默认缓冲区大小可能太大并导致 Pod 故障。减少负载的一种方法是降低记录器的刷新间隔。这可以防止日志溢出缓冲区。你还可以添加更多刷新线程来处理大量日志试图同时填充缓冲区的情况。
|
||||
|
||||
+13
@@ -79,6 +79,19 @@ Rancher Logging 有两个角色,分别是 `logging-admin` 和 `logging-view`
|
||||
|
||||
## 故障排除
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### 日志缓冲区导致 Pod 过载
|
||||
|
||||
根据你的配置,默认缓冲区大小可能太大并导致 Pod 故障。减少负载的一种方法是降低记录器的刷新间隔。这可以防止日志溢出缓冲区。你还可以添加更多刷新线程来处理大量日志试图同时填充缓冲区的情况。
|
||||
|
||||
+13
@@ -79,6 +79,19 @@ Rancher Logging 有两个角色,分别是 `logging-admin` 和 `logging-view`
|
||||
|
||||
## 故障排除
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### 日志缓冲区导致 Pod 过载
|
||||
|
||||
根据你的配置,默认缓冲区大小可能太大并导致 Pod 故障。减少负载的一种方法是降低记录器的刷新间隔。这可以防止日志溢出缓冲区。你还可以添加更多刷新线程来处理大量日志试图同时填充缓冲区的情况。
|
||||
|
||||
+13
@@ -79,6 +79,19 @@ Rancher Logging 有两个角色,分别是 `logging-admin` 和 `logging-view`
|
||||
|
||||
## 故障排除
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### 日志缓冲区导致 Pod 过载
|
||||
|
||||
根据你的配置,默认缓冲区大小可能太大并导致 Pod 故障。减少负载的一种方法是降低记录器的刷新间隔。这可以防止日志溢出缓冲区。你还可以添加更多刷新线程来处理大量日志试图同时填充缓冲区的情况。
|
||||
|
||||
+13
@@ -79,6 +79,19 @@ Rancher Logging 有两个角色,分别是 `logging-admin` 和 `logging-view`
|
||||
|
||||
## 故障排除
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### 日志缓冲区导致 Pod 过载
|
||||
|
||||
根据你的配置,默认缓冲区大小可能太大并导致 Pod 故障。减少负载的一种方法是降低记录器的刷新间隔。这可以防止日志溢出缓冲区。你还可以添加更多刷新线程来处理大量日志试图同时填充缓冲区的情况。
|
||||
|
||||
@@ -116,6 +116,19 @@ By default, Rancher collects logs for control plane components and node componen
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### The Logging Buffer Overloads Pods
|
||||
|
||||
Depending on your configuration, the default buffer size may be too large and cause pod failures. One way to reduce the load is to lower the logger's flush interval. This prevents logs from overfilling the buffer. You can also add more flush threads to handle moments when many logs are attempting to fill the buffer at once.
|
||||
|
||||
@@ -116,6 +116,19 @@ By default, Rancher collects logs for control plane components and node componen
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### The Logging Buffer Overloads Pods
|
||||
|
||||
Depending on your configuration, the default buffer size may be too large and cause pod failures. One way to reduce the load is to lower the logger's flush interval. This prevents logs from overfilling the buffer. You can also add more flush threads to handle moments when many logs are attempting to fill the buffer at once.
|
||||
|
||||
@@ -116,6 +116,19 @@ By default, Rancher collects logs for control plane components and node componen
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### The Logging Buffer Overloads Pods
|
||||
|
||||
Depending on your configuration, the default buffer size may be too large and cause pod failures. One way to reduce the load is to lower the logger's flush interval. This prevents logs from overfilling the buffer. You can also add more flush threads to handle moments when many logs are attempting to fill the buffer at once.
|
||||
|
||||
@@ -116,6 +116,19 @@ By default, Rancher collects logs for control plane components and node componen
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### The Logging Buffer Overloads Pods
|
||||
|
||||
Depending on your configuration, the default buffer size may be too large and cause pod failures. One way to reduce the load is to lower the logger's flush interval. This prevents logs from overfilling the buffer. You can also add more flush threads to handle moments when many logs are attempting to fill the buffer at once.
|
||||
|
||||
@@ -116,6 +116,19 @@ By default, Rancher collects logs for control plane components and node componen
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Resource Exhaustion of `inotify` Watchers and File Descriptors
|
||||
|
||||
When enabling the **Logging** app on Linux systems that heavily monitor the filesystem, you may encounter `Too many open files` or `CrashLoopBackOff` failures related to applications that leverage `inotify` to watch for file changes.
|
||||
|
||||
This happens because the Linux kernel caps the number of files a user can open and the number of directory paths a subsystem can watch simultaneously. To resolve this, you must explicitly increase your `inotify` system limits.
|
||||
|
||||
Below are example commands an admin user can run to increase system limits for `inotify` user instances and watches:
|
||||
|
||||
```shell
|
||||
sysctl -w fs.inotify.max_user_instances=8192
|
||||
sysctl -w fs.inotify.max_user_watches=524288
|
||||
```
|
||||
|
||||
### The Logging Buffer Overloads Pods
|
||||
|
||||
Depending on your configuration, the default buffer size may be too large and cause pod failures. One way to reduce the load is to lower the logger's flush interval. This prevents logs from overfilling the buffer. You can also add more flush threads to handle moments when many logs are attempting to fill the buffer at once.
|
||||
|
||||
Reference in New Issue
Block a user