ChatGPT解决这个技术问题 Extra ChatGPT

Ignore outliers in ggplot2 boxplot

How would I ignore outliers in ggplot2 boxplot? I don't simply want them to disappear (i.e. outlier.size=0), but I want them to be ignored such that the y axis scales to show 1st/3rd percentile. My outliers are causing the "box" to shrink so small its practically a line. Are there some techniques to deal with this?

Edit Here's an example:

y = c(.01, .02, .03, .04, .05, .06, .07, .08, .09, .5, -.6)
qplot(1, y, geom="boxplot")

https://i.stack.imgur.com/mb3In.png

Some sample data and a reproducible example will make it easier to help you.
my file is 200 meg! Just take any dataset where there are lots of datapoints between the 1st and 3rd quantile and a few outliers (you only need 1). If the outlier is far away from the 1st/3rd then necessarily the boxes are going to shrink to accomodate the outlier
Yes, that's what I had in mind. Make up such a dataset and use dput() to post it here together with the ggplot() statement you use. Help us to help you.
Can't you just alter the y-axis limits to "zoom" in on the part of the y-axis you're interested in?
let me look.... Oh yes, sorry. Just do fivenum() on the data to extract what, IIRC, is used for the upper and lower hinges on boxplots and use that output in the scale_y_continuous() call that @Ritchie showed. This can be automated very easily using the tools R and ggplot provide. If you need to include the whiskers as well, consider using boxplot.stats() to get the upper and lower limits for the whiskers and use then in scale_y_continuous().

R
Richie Cotton

Use geom_boxplot(outlier.shape = NA) to not display the outliers and scale_y_continuous(limits = c(lower, upper)) to change the axis limits.

An example.

n <- 1e4L
dfr <- data.frame(
  y = exp(rlnorm(n)),  #really right-skewed variable
  f = gl(2, n / 2)
)

p <- ggplot(dfr, aes(f, y)) + 
  geom_boxplot()
p   # big outlier causes quartiles to look too slim

p2 <- ggplot(dfr, aes(f, y)) + 
  geom_boxplot(outlier.shape = NA) +
  scale_y_continuous(limits = quantile(dfr$y, c(0.1, 0.9)))
p2  # no outliers plotted, range shifted

Actually, as Ramnath showed in his answer (and Andrie too in the comments), it makes more sense to crop the scales after you calculate the statistic, via coord_cartesian.

coord_cartesian(ylim = quantile(dfr$y, c(0.1, 0.9)))

(You'll probably still need to use scale_y_continuous to fix the axis breaks.)


So I would have to calculate the lower/upper - perhaps by calculating the 1st/3rd percentile? Meaning there's no auto-magic way to tell gg-plot2 to ignore outliers and scale intelligently?
Be careful with scale_y_continuous(limits=...) This will remove data that fall outside the limits and then perform the statistical calculations. In other words the mean and other summaries will be affected. If this is what you want, then great. The alternative is to use coord_cartesian(limits=...) - this 'zooms' in without removing data or affecting the summaries.
@Andrie - thanks! I don't want mean and other summaries to be affected.
coord_cartesian() does not play well with coord_flip(), in my experience, so I prefer scale_y_continuous().
This is the best solution. The reason I want to hide outliers is because I am also plotting jittered points with geom_jitter. In this case the outliers just get in the way and make it look like there are more points than there should be.
R
Ramnath

Here is a solution using boxplot.stats

# create a dummy data frame with outliers
df = data.frame(y = c(-100, rnorm(100), 100))

# create boxplot that includes outliers
p0 = ggplot(df, aes(y = y)) + geom_boxplot(aes(x = factor(1)))


# compute lower and upper whiskers
ylim1 = boxplot.stats(df$y)$stats[c(1, 5)]

# scale y limits based on ylim1
p1 = p0 + coord_cartesian(ylim = ylim1*1.05)

+1 for automatic computation, +1 for using coord_cartesian to zoom rather than excluding data
@Ben - you have two accounts? =) @Ramnath - this is a really elegant solution
Using the above method, limits might get biassed by a small extreme on one side and and big extreme on the other, e.g. ylim <- c(-0.1, 1000) * 1.05 gives [1] 0.105 1050. To get equal limits around the mean you could use ylim + c(-0.05, 0.05) * diff(ylim) / 2. Prettier in my opinion.
@Ramnath what does the $stats[c(1,5)] do?
The is not working if you use facet_grid(). Then you have multible boxplots instead of one. Thus you don't get the right limits.
M
Max Ghenis

I had the same problem and precomputed the values for Q1, Q2, median, ymin, ymax using boxplot.stats:

# Load package and generate data
library(ggplot2)
data <- rnorm(100)

# Compute boxplot statistics
stats <- boxplot.stats(data)$stats
df <- data.frame(x="label1", ymin=stats[1], lower=stats[2], middle=stats[3], 
                 upper=stats[4], ymax=stats[5])

# Create plot
p <- ggplot(df, aes(x=x, lower=lower, upper=upper, middle=middle, ymin=ymin, 
                    ymax=ymax)) + 
    geom_boxplot(stat="identity")
p

https://i.stack.imgur.com/vQwX4.png


D
Dirk Eddelbuettel

One idea would be to winsorize the data in a two-pass procedure:

run a first pass, learn what the bounds are, e.g. cut of at given percentile, or N standard deviation above the mean, or ... in a second pass, set the values beyond the given bound to the value of that bound

I should stress that this is an old-fashioned method which ought to be dominated by more modern robust techniques but you still come across it a lot.


Whoever just downvoted silently: leave comment to explain the why.
Wasn't me. Just wanted to add that having whiskers that stop at percentiles (usually 10th and 90th) seems to be very common with environmental data.
I was a silent +1, and wish I had another to offer. Winsorizing is almost always done in econ + finance. If SFun has outliers that ruin data visualiation, I wonder what is their effect on data analysis.
was re-reading this post, you mentioned that windsorizing is an older technique....what would be some more modern techniques?
In general, robust methods as a development of the last 30+ years.
D
Dongdong Kong

gg.layers::geom_boxplot2 is just what you want.

# remotes::install_github('rpkgs/gg.layers')
library(gg.layers)
library(ggplot2)
p <- ggplot(mpg, aes(class, hwy))
p + geom_boxplot2(width = 0.8, width.errorbar = 0.5)

https://i.stack.imgur.com/ZQpAV.png


Thanks!! Tested with my data, working perfecty! I would recommande this solution, although I am not sure about the stability / long time support of github things.
Hi @Gildas, this is a long-term supported package, which is a package I used everyday, github.com/rpkgs/Ipaper.
I
IggyM

If you want to force the whiskers to extend to the max and min values, you can tweak the coef argument. Default value for coef is 1.5 (i.e. default length of the whiskers is 1.5 times the IQR).

# Load package and create a dummy data frame with outliers 
#(using example from Ramnath's answer above)
library(ggplot2)
df = data.frame(y = c(-100, rnorm(100), 100))

# create boxplot that includes outliers
p0 = ggplot(df, aes(y = y)) + geom_boxplot(aes(x = factor(1)))

# create boxplot where whiskers extend to max and min values
p1 = ggplot(df, aes(y = y)) + geom_boxplot(aes(x = factor(1)), coef = 500)

https://i.stack.imgur.com/PTCJp.png

https://i.stack.imgur.com/pjgdb.png


R
Roberto Visintainer

Simple, dirty and effective. geom_boxplot(outlier.alpha = 0)


Hi, this does not adress the problem of the y scale extending too much. The OP said " I don't simply want them to disappear (i.e. outlier.size=0), but I want them to be ignored such that the y axis scales to show 1st/3rd percentile."
e
eckart

The "coef" option of the geom_boxplot function allows to change the outlier cutoff in terms of interquartile ranges. This option is documented for the function stat_boxplot. To deactivate outliers (in other words they are treated as regular data), one can instead of using the default value of 1.5 specify a very high cutoff value:

library(ggplot2)
# generate data with outliers:
df = data.frame(x=1, y = c(-10, rnorm(100), 10)) 
# generate plot with increased cutoff for outliers:
ggplot(df, aes(x, y)) + geom_boxplot(coef=1e30)

It just extends the whiskers, it doesn't rescale the chart whatsoever

关注公众号,不定期副业成功案例分享
Follow WeChat

Success story sharing

Want to stay one step ahead of the latest teleworks?

Subscribe Now